Governed memory · stale-context lab · methodology

How we tested whether a confident summary stops an agent from checking its facts

The result page states what happened. This page states how, so you can decide whether to believe it: the fixtures, the arms and why those arms, the isolation proof, the gate that refuses a rigged benchmark, two bugs our own grader had, and the parts of this nobody outside can reproduce yet.

results the findings ↗ fixtures 6-fact prose · 20-fact graded · 6-construct code corpus fabricated, publishable host darwin 25.5

01 · the claim under test

An agent checks a fact only if it has a reason to doubt it

A task needs some facts. The always-loaded summary asserts a value for each. Somewhere on disk, a dated note records the value that actually replaced it. Before answering, the agent either verifies — spend a few tool calls, get the true value — or trusts — spend nothing, emit what the summary says.

It verifies only when the expected cost of being wrong exceeds the cost of looking. Both sides of that comparison are the agent's own estimate, and the first one is the problem: a confident, self-consistent summary drives its perceived chance of being stale toward zero, whatever the true rate is. Staleness is not underweighted. It is invisible. We call this the answer-before-doubt barrier, and everything on the results page is a consequence of it.

The operational consequence, which is what makes this testable: a mitigation helps only if it raises the agent's suspicion or lowers the cost of looking. A signal that gives the agent a reason to doubt can trigger recovery. A signal that merely de-rates its own evidence — "this may be unverified" — leaves the summary's authority untouched and can make things worse. Those two predictions point in opposite directions, so the design below tests them against each other rather than only against the floor.

02 · the fixture

A fabricated product, so no model can know the answer already

Every fixture is built around Marlin, a product that does not exist. Its facts and their values are invented, which closes the obvious hole: an agent cannot be scoring well from pretraining, and a stale answer cannot have come from anywhere but the channel we put it in.

The corpus is generated with a filler parameter n: n topically plausible filler notes, plus one dated note per fact recording both the superseded value and the one that replaced it. The two corpus sizes are easy to confuse, so they are stated explicitly wherever they appear.

fixture	what the agent must produce	facts	corpus	graded by
6-fact prose facts.mjs	a product one-pager, using the facts fluidly rather than answering a checklist	6	76 notes (n=70)	keyword oracle with negation and cross-contamination guards
corpus-size control	identical	6	13 notes (n=7)	identical
20-fact graded facts-hard.mjs	answer 20 questions across five trap tiers	20	100 notes	per-tier oracle, oracle-hard.mjs
6-construct code facts-code.mjs	a working client that creates a contract check and polls it	6	46 notes	exact code tokens — create_board() against create_contract()

The prose task deliberately does not ask "what is the pricing?" A checklist manufactures doubt; a one-pager does not. The coding fixture exists because a prose deliverable invites the objection that this is a writing problem: there, the always-loaded file is an API quick-reference stamped "last verified 2026-01-15" asserting six obsolete constructs, and a wrong answer is a call that returns 410.

Where the fixtures came from, and why that is a weakness. The traps were designed with a frontier model that was also under test. The external pilot exists precisely because of that, and the honest position is that a fully independent adversary is still future work. The fact notes also carry topically obvious filenames, which flatters every arm that searches.

03 · the arms

Sorted by who is made responsible, not by what they are called

Most treatments list mitigations as independent tools — RAG, tiered memory, a decision log. That list does not predict anything. Sorting them by which actor is made responsible for the trusted context being correct, and when does: each family fails exactly where its actor defaults, and the two stressors on the results page are not a grid we picked, they are the two ways an actor defaults.

class	arm	responsible actor	fails when
C0	floor — the stale summary, nothing else	nobody	always; it is the baseline
C1	verify — one line: check the notes first	the agent, on doubt	the model declines to search
C2	RAG — retrieve from the raw notes	the agent, on doubt	the model declines to search
C3	TierMem — escalate when the summary looks insufficient	the agent, on insufficiency	the summary is complete and wrong
C4	Memory Bank · ADR · spec — a curated store	a human, at write time	maintenance lags
C4′	CUPMem — adjudicate at write time	a model, at write time	compression happens before the question
C5	mla push — the in-force values delivered into context	the system, at delivery	coverage or precedence fails
—	no-summary — strip the assertions out	—	not deployable; it isolates the cause

The two stressors follow directly. A weaker model tests whether the C1–C3 actor will act. Maintenance lag — the curated store missing the two most recent supersessions, while the raw notes still carry them — tests whether the C4 actor has. A study that varies neither reports a tie, and that tie is an artifact of testing only the conditions in which every actor behaves.

What the C5 arm is, and what it is not — read this before citing any push number. It delivers the fixture's ground-truth current values, already adjudicated, straight into the prompt. Retrieval, conflict resolution and selection are done by construction. That makes it an upper bound on the mechanism: it measures whether a correct, complete, adjudicated delivery gets adopted — not whether any pipeline can produce one, and not the shipping product, which is not in the loop. C1–C3 have to find their own evidence, so their gap to C5 is the combined difficulty of finding, resolving, selecting and adopting. Act II of the results is what happens when the delivery is realistic instead of oracular, and it is much worse.

04 · models, harness, trials

Two harnesses, one context file, and the trial counts that bound each claim

Claude models run under the native claude -p CLI with the fixture's CLAUDE.md auto-loaded. Gemini and OpenAI models run through a REST agentic loop given the same file and list_dir / read_file / write_file tools, graded on the answer.txt it writes, identically. The only push in either harness is the context file itself.

study	trials per cell	what that supports
primary matrix, Claude	3	the floor has zero variance, so N=3 is enough to establish it
primary matrix, Gemini and GPT-5.4	6	pooled over two independent runs
primary matrix, GPT-5.5 and GPT-5.6	3	same as Claude
adoption study (coverage and precedence)	2–3	an existence result only — see the caveat below
Opus 5 floor	12	a rate estimate with a wide interval, against an 8-trial same-build control
coding fixture	3	agreement with the prose fixture, not a rate
external pilot	1	a signature reproducing, not a mean

The adoption cells are the weakest evidence on the site and we would rather say so here than have it found. Two or three trials per cell, on Haiku 4.5 and Opus 4.8, with Opus 5 added to the cells where it was run. They are counterexamples: they show that a delivered correct value can be ignored, which is enough to falsify "delivery is sufficient". They are not a claim about how often that happens.

An artifact gap in one cell. Our contemporaneous ledger records five Haiku trials on the unresolved-conflict cell; three per-trial files survive in runs/, because a re-run wrote over the same filenames. The surviving three agree with the ledger's finding — one scored zero, two produced no deliverable at all — but any count above three on that cell rests on the ledger, not on a file you can re-grade. We report the surviving artifacts.

05 · isolation

Proving the clean room, four ways, because one way is a claim

Every arm runs in a subprocess whose only project instruction is the fixture's own CLAUDE.md. That is easy to assert and worth very little unless it is checked, so it is checked structurally, by the CLI's own report, and from both sides with a canary. All seven checks pass on both Claude models; the REST harnesses expose nothing but the injected context.

check	what it rules out
structural — the config directory is bare	no global context file, no hooks, no access to our internal tooling
init event — the CLI's own startup record	0 servers connected, 0 mcp__meetless* tools available. The system's report, not the model's claim
positive canary — a unique marker in the arm's own file	proves the fixture's instructions actually loaded, rather than nothing loading at all
negative canary — six secret markers from our real environment	none appear in any output; no private tool name, server name or product term leaked in

Re-run it yourself with validate-isolation.mjs. The logs from our runs are kept.

06 · the anti-tautology gate

The check that exists because three of our earlier benchmarks failed it

The easiest way to win a benchmark is to build one whose answer lives only in the channel your own arm holds. We did that three times without noticing. One retrieval arm scored 3/3 on a corpus the answer had been stripped from, which is not a result, it is a leak.

All three are retracted, and derivability.mjs now refuses any benchmark whose graded answer is not recoverable from every arm's corpus. Without that gate, the winner is decided before a model runs.

The same discipline applied to the strongest rival. The CUPMem store is built by a model reading the raw notes, via cupmem-adjudicate.mjs. Hand-writing it would have made it correct by construction and proved nothing.

07 · what we count

Three numbers, because accuracy alone hides which family failed

Accuracy — facts at their current value, by a deterministic oracle. No LLM judge anywhere in the grading path.
Stale rate — facts emitted at the superseded value. This is a commission: the agent stated something false.
Documents read — the behavioural diagnostic. It is what separates "looked and got it wrong" from "never looked", and it is the whole reason the floor result means what it means.

Collapsing "wrong" and "missing" into one number destroys the finding. A miss that is an omission is a system that declined to answer; a miss that is a commission is a system that asserted a superseded value into a deliverable. Across every push run in this study the misses are omissions and the count of commissions is zero, while every other arm's misses are stale values. Accuracy alone cannot see that difference.

Two bugs our own oracle had, both found after publishing numbers.

1. Cross-contamination. A stale tagline containing the words "marketing team" contaminated the separate audience fact. The scorer now strips tagline signatures before grading the audience, and six saved cells were re-graded from their immutable stored answers by regrade.mjs.

2. A false negative in the code grader. Models write DEFAULT_TIMEOUT = 30.0 and then pass timeout=DEFAULT_TIMEOUT, which a case-sensitive timeout=30 pattern scored as absent despite being correct. Every trial was re-graded from stored output. It moved the coding floor result against our hypothesis, which is the direction that makes it worth reporting: the strongest model scored better than we had it. This is the failure mode a keyword oracle has, and it is the price of not using a judge.

08 · threats

The arguments we would make against this if it were someone else's

Low trial counts. Means and per-trial vectors, no significance testing. The floor claim survives that because its variance is zero; the adoption claims are existence results and the Opus 5 rate is a point estimate with a wide interval.
The fixtures are ours. Designed with a model that was also under test. The external pilot on independently authored scenarios is the answer to this, and it is a pilot: a deterministic first-40 slice, not a random sample.
Cross-vendor cost is not comparable. The two harnesses differ, so token and cost comparisons are made within a vendor only. Cost is absent entirely for every Claude CLI run and every GPT-5.5 and GPT-5.6 run, so no study-wide cost superlative is establishable at all.
The push arm is a mechanism, not a product. Oracle-selected payload, placed in the prompt rather than fired by a production hook. It bounds what a perfect delivery could achieve. It does not measure anything we ship.
Refusal scores as zero. Our oracle grades an abstention as zero accuracy, so this task cannot reward declining to commit. Where abstention is acceptable in your deployment, an unresolved signal may be far safer than a confident stale answer — that ordering is a judgement about your deployment, not a constant we measured.
Single-turn. We observe non-adoption at one turn. Establishing a delay — how many turns a correct value sits unused — needs a multi-turn design we have not built.

09 · reproducibility

What an outsider can check, and the row they cannot

claim	reproducible?
The barrier itself — a confident stale summary suppresses the search: 0/6, zero tool calls	Yes. A folder of fabricated notes, a summary file, and a plain claude -p run. No product involved; one arm takes a couple of minutes.
The comparisons — Memory Bank, ADR, spec, RAG, TierMem, CUPMem	Yes. Same shape. The CUPMem store is rebuilt by a script, so you can confirm it was not hand-written to lose.
The adoption cells — coverage and precedence	Yes, with the artifact gap noted in section 04 on one cell.
The mla push rows	No — and not because of the repository. That arm does not run mla. It is a fixed block of current facts placed in the prompt, standing in for what a delivery mechanism would supply. Installing mla and activating a folder would not reproduce these rows, because the product is not in the loop.

git clone https://github.com/Meetless/stale-context-bench
cd stale-context-bench
node validate-isolation.mjs --model claude-opus-4-8   # prove the clean room
node matrix.mjs --trials 3 --conc 3                   # every arm x {opus, haiku}
OPENAI_API_KEY=... node openai-matrix.mjs --trials 3  # cross-vendor
node run.mjs --arm hard-push --model claude-opus-4-8 --trials 2   # 20-fact graded fixture
node run.mjs --arm mf-fr-correction6 --model claude-haiku-4-5     # an adoption cell
node cupmem-adjudicate.mjs --arm arms/hard-cupmem     # rebuild the CUPMem store, then run it

The honest summary: the phenomenon is the reproducible part, the product claim is not. The barrier, and the ranking of mitigations against it, rest on fixtures anyone can rebuild. The rows labelled mla push measure an idea — correct values present in trusted context — and until that arm runs the real product end to end they should not be cited as evidence about the product.

fabricated fixtures — no internal data · back to the results ↗