How we tested whether an agent can reconstruct a decision history

A decision history is the changes that stuck — not every change proposed

Over a quarter, a team tunes one knob five times. Twice, a value is floated and then killed — a road not taken. Three times, a value is actually adopted. Months later someone asks the honest question: "what did we set it to, in order?" The right answer is the three that stuck, oldest to newest. The two that were rejected must be left out.

Each change was written down the same way — a dated note that states a value as if setting it. Nothing in the text of a rejected proposal says "this one didn't happen." So the corpus contains five plausible, dated, equally-worded values, and only three of them were ever in force.

H. Sorting five dated notes is trivial for any capable agent. The hard part is exclusion — knowing which two to drop. An agent with the accept/reject verdicts reconstructs the in-force history exactly; an agent with only the raw notes cannot tell an adopted value from a killed one, so it must either decline (honest, useless) or guess (and risk shipping a road not taken as if it were policy).

One variable — the verdict — and a reconstruction, not a lookup

The task asks the agent to reconstruct the in-force history of a single production setting (a rollback window, in minutes), oldest change to newest, writing one integer per line. It is deliberately a reconstruction, not a single-fact lookup: the difficulty is not finding a value, it is deciding which values belong and in what order.

Why this measures the verdict and nothing else

Every note carries its own date, so ordering is free — sort by date and you are done. We deliberately removed every other cue that could leak which values were adopted:

• Uniform wording. Every note — adopted or rejected — uses the same sentence: "Set the production rollback window to N minutes." A rejected proposal does not announce itself.
• No cross-references. No note says "changed from the previous 80" or "reverting 89." Each is an independent point-in-time assertion, the way config-change notes, PR bodies, and changelog fragments actually accrue across months and authors.
• Distinct values, random filenames, shuffled ingest. The five values are distinct two-digit integers; filenames are random hex; ingest order is shuffled. Nothing about position, name, or value hints at status.
• The reject lives only in the verdict. The one and only signal that separates the two killed values from the three adopted ones is the governance accept/reject decision. That is the single variable under test.

Five dated changes, two of them roads not taken, seeded through the real API

1. generate  5 distinct two-digit values on 5 dated slots (~92 days apart,
             oldest -> newest); pick 2 at random to be REJECTED (roads not
             taken) and 3 to be ACCEPTED (the in-force history)
2. write     each as an identically-worded dated note:
               "Set the production rollback window to N minutes."  __
             (adopted and rejected notes are indistinguishable as text)
3. provision a throwaway workspace; ingest ALL five via the real API,
             in shuffled order, under random hex filenames
4. govern    ACCEPT the 3 adopted values · REJECT the 2 roads-not-taken
             (by value regex, so the verdict binds to the number, not the file)
5. verify    governed retrieval SERVES all three accepted values
             -> if any is missing, ABORT (fail-closed; the benchmark is hollow otherwise)
6. on disk   the same five notes go under notes/ for the ungoverned arm (no verdict)
7. inject    the accepted sequence + the rejected values into the oracle's env,
             then DELETE them from the harness's own memory

The accepted sequence and the rejected values are generated per run and live only in the store's DB and the launcher's memory — never in a file the agent can read. The static arm's changelog is the one place the in-force sequence is written in prose; that is deliberate — it is the ceiling we are trying to match.

A deterministic oracle that separates three ways to fail

No model grades this. The agent writes its answer as one integer per line; a shell oracle extracts the integer-only lines, in order, and compares the sequence to the accepted history. Averaging the failure modes together would hide the most interesting result, so the oracle names each one:

Grading the reconstructed sequence (not a single fact) is the point: it measures whether the agent could rebuild "what we actually decided" — the artifact a new engineer, an auditor, or a next-sprint planner actually needs.

Total separation, on every history

Across 3 independently-generated histories × 3 trials, the split was identical every time: static and mla each rebuilt the exact in-force sequence 3/3; the ungoverned arm rebuilt it 0/3. When it failed it did so in one of two ways — 7 of 9 it returned all five values in correct date order (nailing the sort, failing the exclusion), and 2 of 9 it wrote an empty answer rather than invent a history. It was never right, and it was slower being wrong (~267s) than mla was being right (~104s). The ablation is backed by the stored health signal on every trial: mla grounded (queried the reviewOutcome verdict; retrieve 7–12 per trial), static & ungoverned unused (no governed retrieval).

The strongest arguments against this

• "The rejections were never written down — of course the ungoverned arm can't find them." This is the load-bearing modeling choice, and we state it plainly: in this fixture a rejected proposal leaves no note saying "we killed 89"; the reject exists only as the governance verdict. If the rejection were narrated in the corpus, the ungoverned arm could read it and likely pass. We model the common case where a decision is made but not narrated back — the meeting happened, the verdict was reached, and nobody wrote "we rejected the other option" into the notes. That is precisely the gap a governance layer exists to close: it records the accept/reject decision as first-class data instead of trusting someone to remember it.
• "You engineered the notes to be indistinguishable." Yes — that is what an un-narrated decision history is. If the adopted and rejected notes differed in text, the text would carry the verdict and there would be nothing to govern.
• "Small N." A mechanism demonstration on one machine and one model, not a population estimate. The interesting quantity is not the pass rate but which failure mode each arm falls into, and that is legible in every trace.
• "A different prompt would let the ungoverned arm guess correctly." Guessing the exact 3-of-5 subset and the order is a 1-in-many shot; a prompt that pushes it to guess raises the "shipped a road not taken" rate, which is worse, not better.
• Substrate honesty. Runs use a neutral, domain-matched repo with no Meetless code or governance vocabulary, so nothing in the surrounding files cues the answer.

Scope

• Not that a capable agent is stupid — the honest arm's refusal to invent a history is the correct move. The claim is that correctness-when-it-answers requires the verdict; without it the agent is stuck between declining and guessing.
• Not that governance is free. Our retrieval benchmark publishes the token overhead on tasks where memory adds nothing.
• Not the same result as contested-canonical (one contested name, agent ships the dead one) or trust-containment (two byte-identical claims, agent gets stuck). Here the challenge is exclusion across a set — reconstructing which subset of proposals actually took effect. Same verdict mechanism, a third distinct failure mode.
• Not a claim about any competitor's product. The opponent is the general shape of ungoverned memory — a store that remembers every proposal without remembering which ones were adopted.

Run it yourself

cd tools/bench-mla
node run.mjs --tasks decision-history \
  --arms static,ungoverned,mla --trials 3 --substrate neutral

# the 5 values, which 2 are rejected, and the dates are generated per run;
# yours will differ from ours. What should reproduce is the FAILURE MODE:
# ungoverned cannot separate adopted from rejected; mla reconstructs the sequence.

Requires a local Meetless stack and Claude Code with the mla plugin. The harness verifies the fixture is correctly governed (all three accepted values are served by governed retrieval) before any trial and aborts otherwise; it marks a trial invalid — never a loss — if the backend was unhealthy or the provider refused the run. Raw transcripts are not published — they contain the generated values.