The setup, why the reviewer id is unguessable, why the losing arm loses structurally, and the four layers of our own product that were silently dropping the answer.
Every governance product can tell you what was decided. The question an engineer is actually asked at review time is different: "who signed off on this, and did they sign off before or after the API changed?" A decision without an author is not an audit trail, it is a rumour.
So the task asks for three fields — the decided value, the reviewer's identifier, and the approval date — and the oracle grades each one independently. That separation is what makes the result legible: it lets us see an arm get the decision perfectly right and the provenance perfectly absent, which is exactly what happens.
The static arm is the steelman: the team wrote the decision down and put it in CLAUDE.md, in front of the agent, before it started. It recovers the decided value 5 out of 5 times.
Then it is asked who approved it. There is no answer in the file — because a markdown file has no field for a reviewer or a timestamp. You can write one by hand, and teams do, and it goes stale the moment someone changes their mind and forgets to edit the header. The failure is not a model limitation and no better model closes it.
We predicted the static arm would fabricate a plausible approver and date — the kind of confident invention an auditor would have believed. It never did. It said UNKNOWN, honestly, five times out of five. Zero fabrications. The honest finding is cleaner than the one we went looking for: the model is not careless, the file is empty.
The value, the reviewer, and the timestamp are all generated per run. The reviewer identifier is a cuid minted when the fixture accepts the claim through the product's real review API — something like c00m…xxxxxxxxxxxxxxxxxxxx. It exists in the database and in the launcher's memory, and in no file, git object, or environment variable any arm can reach.
That is what makes the grading unfalsifiable: an agent that returns that string did not deduce it, infer it, or get lucky. It read an audit trail. There is no other way to produce it.
The first run scored governed memory the same as the file: who=UNKNOWN, when=UNKNOWN. The agent called retrieval six times, reported the value correctly, and still could not name the reviewer.
The audit trail was never missing. It was written to the claim by the review API and then dropped, silently, by four layers in a row:
| Layer | What it did |
|---|---|
| the claim-axis probe | returned the verdict and not its author |
| the metadata projection | a whitelist — it forwards named keys and silently drops the rest |
| the evidence DTO | had nowhere to put a reviewer |
| the tool contract | never told the model the field existed at all |
Not one of them raised an error. A whitelist does not fail when you forget a key — the field is simply, quietly, not there. We fixed all four and re-ran; the published 4/5 measures the fixed product. The benchmark found the gap, the fix closed it, and the benchmark now measures the fix. That is worth more to us than a number that had been right the first time.
tools/dev-stack.sh up # control + worker + intel, health-gated tools/bench-mla/preflight.sh # refuses to run against a lying stack node tools/bench-mla/run.mjs \ --tasks governed-provenance \ --arms vanilla,static,mla \ --trials 5
The preflight is not decoration. It refuses to benchmark when the worker's queue is backing up, when intel is serving from the wrong virtualenv, when the generated Prisma client has drifted from the schema, or when the machine can still fall asleep mid-run — every one of which produced a plausible, wrong number before it was caught.