We test one claim: a coding agent with governed memory can act on a team decision that exists only in the shared knowledge base, where a memory-less agent cannot. On a knowledge task whose answer lives solely in the governed store, mla passed 5/5 and vanilla passed 0/5 โ vanilla searched the filesystem, guessed, and gave up, spending more tokens and time to remain wrong. On a pure-coding control needing no team knowledge, both passed 5/5, and mla cost +2.1k tokens (+58%) and +17 s for no benefit โ the overhead we publish rather than hide.
The headline is not the numbers; it is the isolation. A benchmark is only as honest as the walls around the memory-less arm. We document seven ways our own harness leaked the answer to "vanilla," how each was sealed, and how you can verify the seal. We are also explicit about what this does not show: the opponent here is a memory-less floor, not a steelman. We owed a real one โ a competent CLAUDE.md snapshot โ and have since built and published it: it loses 0/5 once a decision is superseded. An ungoverned RAG index over the same corpus is the remaining opponent.
The governed-memory thesis is a nine-step loop (observe โ gate โ promote โ share โ detect conflicts โ inject โ monitor โ enforce โ update). This report isolates and tests one link: inject only relevant, approved decisions into a task.
Everything else in the loop โ trust gating, live conflict detection, deterministic enforcement, temporal validity โ is out of scope for this run and is not claimed here.
| Arm | Definition |
|---|---|
| vanilla | Claude Code, headless, no memory layer. Sandboxed so mla's installation and the local store are unreachable โ i.e. a machine where mla was never installed (see ยง4). |
| mla | Claude Code + the mla plugin (governed-memory MCP server + capture hooks), bound to a seeded workspace. Unsandboxed โ it legitimately requires its own installation; that is the treatment. |
| Model | claude-opus-4-8 |
| Claude Code | 2.1.197 |
| mla | 0.2.14 |
| Invocation | claude -p --output-format stream-json --permission-mode bypassPermissions |
| Repository state | identical detached git worktree at a fixed base commit, per trial |
| Design | 2 tasks ร 2 arms ร 5 trials = 20 runs; fresh worktree per trial |
Every arm gets the same repository state, task prompt, model, model configuration, permission mode, tool availability, and time budget. The only intended difference is the memory mechanism. The vanilla sandbox (ยง4) removes access to mla's binary/credentials and the local store โ things a genuinely mla-free machine would not have โ and does not restrict repository reads, file writes, or the model API. It is a deprivation that matches the control condition, not a handicap.
The agent is asked to report three organizational facts about an internal pilot โ a codename, an exit criterion, and a report owner. These exist only in the seeded governed knowledge base: not in the repository code, git history, environment, on-disk notes, or any reachable service except through mla. mla can retrieve them via the governed-memory MCP tool; vanilla has no path to them and should answer UNKNOWN.
The facts are invented at run time, never authored: a random codename, a random exit percentage, and a random owner name, seeded into a throwaway workspace through the same public API routes a customer uses (add โ accept), then verified retrievable by polling the store before any agent starts. They live in exactly two places โ the store's database and the launcher's process memory โ and touch no file, commit, or environment variable the agent can read.
This closes the objection that isolation alone cannot: the answer is younger than the run. It cannot be in any model's training data, cannot have leaked into this repository, and differs on every execution โ so a published result can never contaminate a future one. It also means your numbers will differ from ours; what should reproduce is the separation, not the literal values.
Write a slug() string utility to an exact specification with worked examples. Fully self-contained; requires no team knowledge. This isolates governed memory's overhead when it provides no benefit.
Grading runs after the agent process exits. Task A's oracle checks the agent's answer.txt against expected values injected as environment variables at launch โ no literal answer is ever written to a file the agent could read. Task B's oracle imports the produced module and asserts it against the spec's worked examples. Both are pure, deterministic checks; efficiency metrics are computed only over runs that passed, so a fast-but-wrong run is a failure, never an "efficiency win."
A knowledge benchmark is trivially gamed if the memory-less arm can reach the answer another way. Under bypassPermissions, the agent is filesystem- and loopback-omniscient, so obscurity is not enough. We enforce a hard invariant.
The answer to a governed-knowledge task exists in exactly two places, neither reachable by the vanilla arm: the governed-store database, and the launcher's process memory. It exists in no file, git object, environment variable, or network endpoint the agent can reach.
Getting there took seven iterations. Each was found by forensics on a suspicious pass, then sealed. We publish them because the failures are the credibility.
| # | How "vanilla" reached the answer | Seal |
|---|---|---|
| 1 | Task's own oracle/spec files sat in the worktree | strip the benchmark dir from the worktree |
| 2 | Ran git show <sha>:oracle against shared history | sever .git โ the worktree becomes a plain dir |
| 3 | Grepped the facts from a sibling notes vault on disk | keep facts in the DB only; never on disk |
| 4 | The oracle script itself contained the literal answer | inject expected values via env; no literals on disk |
| 5 | Dumped env and read the expected values | delete them from the child env; re-inject only into the grader |
| 6 | Inherited a global memory MCP server + mla on $PATH | strip both from the vanilla config and PATH |
| 7 | Found the binary by absolute path, read its config, and queried the live store over localhost | sandbox: deny file-read of the install + deny all loopback egress |
sandbox-exec, sanitized)(version 1) (allow default) ; deny reading mla's install (binary + credentials + store URL) (deny file-read* (subpath "$MLA_HOME")) ; ~/.meetless ; deny reading the project tree: store env-keys, notes, and the bench oracle (deny file-read* (subpath "$WORKSPACE_TREE")) ; deny ALL loopback egress: intel, control, Postgres, Weaviate (deny network-outbound (remote ip "localhost:*"))
The external model API stays reachable; the worktree and the grader's own config dir live outside the denied subtrees, so the task itself is unaffected. Verified: under this profile the agent can read the worktree and reach the model, but every attempt to read the mla install or open any local port returns EPERM/refused.
This is a stopgap for benchmarking inside the mla repository against a live local store โ two conditions that make leakage possible in the first place. The durable design (next phase) is a neutral task substrate that contains no mla client code, plus a store reachable only through mla's auth boundary, so the memory-less arms have nothing to learn from and nothing to connect to โ making most of the sandbox unnecessary.
Per run we record: task success (deterministic oracle), filesystem searches (grep/glob/find + grep-equivalent shell), work tokens (input+output, excluding cache), wall-clock time, turns, and governed-rule violations (independent post-hoc checker, same on both arms). We report medians with a nonparametric effect size (Cliff's ฮด).
This is a pilot. p-values are normal-approximation and directional only; we lead with pass-rate and effect size, not significance. We make no claim of statistical superiority at this N. A published version needs larger N, paired per-task bootstrap deltas, and cost-per-successful-run alongside cost-per-run.
| metric | mla | vanilla |
|---|---|---|
| passed | 5/5 โ | 0/5 โ |
| retrieval health | grounded | none |
| searches (median) | 0 (every trial) | 9 (6โ15) |
| work tokens (median) | 5.4k | 21.2k (โค24k) |
| context tokens (median) | 163k | 1.12M (6.9ร) |
| agent time (median) | 31 s | 4.7 min (โค7.2) |
Vanilla's figures are the cost of failing: it spent more and produced the wrong answer. With no passing vanilla runs, the comparison reduces to the only fact that matters here โ one arm can answer the question at all, and the other cannot.
| metric | mla | vanilla |
|---|---|---|
| passed | 5/5 โ | 5/5 โ |
| searches (median) | 0 | 0 |
| work tokens (median) | 5.8k | 3.7k |
| agent time (median) | 31 s | 14 s |
| overhead | +2.1k tokens (+58%) ยท +17 s ยท Cliff's ฮด = 1.00 | |
Perfect separation on tokens: every mla run cost more than every vanilla run. This is the price of the always-on grounding floor on a task that needs none of it โ reported, not hidden.
CLAUDE.md snapshot โ accurate the day it was written โ and the snapshot loses 0/5 once the decision is superseded, reporting the stale value or never answering. The remaining opponent, an ungoverned RAG index over the same corpus with governance as the sole variable, is still owed.The harness reads task prompts, oracles, and metrics from the repository; results are aggregate-only (raw transcripts, which embed the answer, stay out of published artifacts and are secret-scanned).
cd tools/bench-mla node run.mjs --trials 5 # tasks ร arms ร trials โ runs/ node analyze.mjs # runs/ โ results/summary.json node report.mjs # โ results/report.md + .html
Reproducibility target is the protocol, not the exact numbers: hosted-model and Claude Code version drift make bit-identical reproduction unrealistic. Run the same protocol; expect statistically comparable results.