DRAFT
These are early results still under review. They have not been announced yet. Numbers may change (Gemini results vary). The most stable findings are for floor (0/6) and mla push (5.0–6.0). We are sharing this for early feedback.
Governed memory · stale-context lab
Your AI coding agent is working from outdated information
A coding agent with a project summary in its context trusts that summary completely. It does not check if the summary is still true. If a decision changes, the agent uses the old one. It does this confidently, without reading any files. We saw this happen with ten models from three vendors, including three successive OpenAI generations. We tested every common way to prevent this. The methods that seem to work only tie under perfect conditions. These conditions never happen in a real project.
Preliminary · internal
10 models · 3 vendors
+ reasoning ablation
11 arms · 3 trials/cell
6 superseded facts
isolation proven
fixtures fabricated + publishable
run date 2026-07-18
claude 2.1.197
models opus-4.8 · haiku-4.5 · gemini-2.5-pro/flash · gpt-5.4/mini · gpt-5.5 · gpt-5.6-sol/luna/terra
mla source build
host darwin 25.5
We reproduced the failure. It happens even with the best models.
0/6stale summary
Opus & Haiku
both make
0tool calls
never looks
·
6/6mla push
Opus & Haiku
We gave an agent a summary file called CLAUDE.md. It contained six facts about a product. All six facts were out of date. The correct facts were in other notes on the disk. We asked the agent to write a one-page summary. Both the strongest and cheapest models wrote the wrong summary. They used only the stale facts and read zero files. This is not a problem with weak models. It is not a problem with finding information. A confident summary in the context stops the agent from checking its facts. If we remove the summary, the agent searches for the right answer and succeeds. The summary is the problem, not the model.
New to methods like Memory Bank, ADR, RAG, or spec-driven? See the methods reference →. Learn what each one is, how to set it up, and what users say.
The result in one table
All methods besides mla push fail for common, real-world reasons
Two things matter most. First, a weaker model. Teams often use cheaper models, not the best ones. Does the method work if the model doesn't try hard to find facts? Second, out-of-date notes. No one keeps their notes perfectly updated. Does the method work when the notes are stale? Every other method failed at least one of these tests. Only one method passed both.
Methods that tell the model to search fail with a weaker model. The model just won't search. Methods that use a special set of notes fail when those notes get old. They always get old. mla push is the only method that works without these problems. It doesn't need a human to update notes. It doesn't need the agent to decide to search. The cost table also shows it uses the least context of any method that gets the right answer.
Panel 1 · perfect maintenance (a fiction)
The methods only tie when using the best models
With Opus, everything seems to tie. But look at the results for Haiku.
With Haiku, verify and RAG fail completely. They made 0 tool calls. We checked that the search tool was working. The right answers were easy to find. The weaker model simply chose not to search. Also, “perfect maintenance” is not realistic. See Panel 2 for why. And note that 6/6 is this test's ceiling: it separates the methods that fail from the ones that don't, but it cannot rank the ones that pass. The last note below shows what a harder test reveals underneath that tie.
Panel 2 · maintenance lag (reality)
Specially prepared notes get stale. Raw notes do not.
The two most recent decisions were missing from the special notes.
Methods using special notes failed on both models. The agent trusts the incomplete notes. It doesn’t check the complete, raw notes. This is the same problem as the stale CLAUDE.md file. Methods that use raw notes don't have this problem. But RAG’s advantage is useless with Haiku, because Haiku never searches.
New — a harder test, because the one above has a ceiling
When every method scores 6/6, the test has run out of room
Look at the Opus column in Panel 1: almost everything ties at 6/6. That tie is not the methods being equal — it is a six-question test being too easy to rank them. So we built a harder one: 20 facts, hidden among 100 notes, across five kinds of trap. Each trap defeats a different shortcut, so the score breakdown tells you which strategy a method is actually running.
TRAP 1 · RECENCY
The value simply changed. Taking the newest note is correct here.
TRAP 2 · REJECTED
The newest note is a proposal that was turned down. The older decision still stands. Taking the newest is wrong.
TRAP 3 · SCOPED
The newest note is an exception for one tier only. The general default did not change.
TRAP 4 · MULTI-STEP
The answer is written in no note at all. It has to be worked out from two of them.
TRAP 5 · NEAR-DUPLICATE
Two nearly identical notes; one is a superseded draft. Only the status marker tells them apart.
No ceiling — the scores now spread across the whole range, and three things appear that the easier test could not show.
1. “Check the notes first” is far less reliable than 6/6 made it look. On identical inputs it scored 19/20 on one run and 10/20 on the next. The bad run read some notes and then just took the newest one every time. The push scored 20/20 both times, with no variation.
2. The weaker model can find the facts but not reason over them. Given the notes with no summary at all, Haiku still scored 0–1 out of 4 on the multi-step trap — it retrieved the right notes and could not work out “three times the previous value.” Opus got 3–4 out of 4.
3. The hardest trap for everyone is the scoped exception — treating a rule that applies to one customer tier as if it applied to everyone. Even the best runs slip on it.
This harder test also cuts against us, and that is the more important half. On it, the stale-summary failure stops happening on the strongest model: faced with 20 questions, Opus decides a 20-line summary obviously can’t cover them, and just reads all 100 notes (in 3 tool calls) — scoring 16/20 without being told to look. On Haiku it still fails completely (0/20). We think this sharpens the finding rather than shrinking it: the failure bites hardest when the stale summary looks like it already covers the job, which is the normal case for a project summary. The two tests measure different things — whether the agent looks, versus whether it can reason over what it finds — and we never mix their numbers.
Not a Claude quirk — it replicates across three vendors
The failure happens with all ten models. The common fix does not.
We tested the most important methods on Gemini 2.5 Pro/Flash and on four OpenAI generations (GPT-5.4 and 5.4-mini, GPT-5.5, and the three GPT-5.6 models). They used the same setup with a summary and file tools. The stale-summary failure happened on every model. The most common fix is to tell the model to “check the notes.” This only worked reliably on six of the ten models. The results were unpredictable. Gemini’s best model failed, but OpenAI’s passed. The only thing that worked every time was keeping the current facts in the context.
The floor method fails on all ten models (0/6, with no variation) — including three straight OpenAI generations. Whatever GPT-5.5 and 5.6 improved, none of it touches this. The verify method works on six of the ten and you can’t predict which. Gemini’s best model ignores the instruction (2.0), while OpenAI’s follows it (6.0). mla push is the only method that works on every model (5.0–6.0). We also tested TierMem, a published two-tier memory that escalates to the raw log only when its summary looks insufficient: it is worse than the one-line “check the notes” instruction on six of the ten models, and better on none. A stale summary is never insufficient — it answers every question, just wrongly. When it scored below 6, it missed a fact. It never used a stale fact. Special notes add another problem. They are mostly reliable. But Gemini Flash read the complete notes and still used stale information (Memory Bank 0/6). So even the methods that seem to tie depend on the vendor.
Is it a reasoning-model artifact? No — and the ablation is telling
Search methods require the model to reason. The push method does not.
OpenAI provides a good way to test this. gpt-5.4 can reason, but gpt-4.1 cannot. Without reasoning, every method that needs to figure things out fails completely. But the push method still works. That's because it gives the model the answer directly. It doesn't ask the model to figure out the answer.
Reasoning drives methods like verify, ADR, and search. Without it, they score 0/6. The methods that still work don't need reasoning. They just copy information from a small set of notes, or use the push. With the push, the answer is already in the context. So, the failure is not a reasoning problem. The fix works even if the model can't reason. (We also tested this with Gemini by turning its thinking on and off. The results were similar but less clear. Gemini sometimes gave no answer at all when its thinking was off (about 1 in 10 tries). So we rely on the cleaner results from the OpenAI models.)
The fourth dimension — accuracy is not the only axis
The one method that always works is also the cheapest and fastest
Getting the right answer is the minimum requirement. But what does it cost? We averaged the costs over 3 trials. The methods that work on Opus cost 5–16× more money. They also take 3–6× more time than the push method. On Haiku, those methods seem cheap only because they get the wrong answer.
Using mla push with Haiku is the cheapest, fastest, and most accurate option we tested. It costs $0.027 and takes 12 s. It makes zero tool calls. This is because the correct facts are already in the context, so it doesn't need to search. To be fair, the push method costs $0.26 on Opus. Opus is just an expensive model, so the push is not free. The no-summary method shows the cost of a common fix. If you remove the summary, even a weak model uses 517k context tokens to get most of the facts right.
Why this number is trustworthy
We proved our test environment was clean for both models
Each test ran in a completely separate environment. The only instructions the agent had came from the CLAUDE.md file in its folder. It had no access to other memory, tools, or project knowledge. We proved this in three different ways. All our checks passed on both Opus and Haiku.
01 · structural
Config dir is bare
There was no global CLAUDE.md file. We removed all special hooks from the settings. The test setup had no access to our internal tools.
02 · init event
The CLI’s own report
The system's own startup log showed it connected to 0 meetless servers. It also showed 0 mcp__meetless* tools were available. This was the system's report, not the model's claim.
03 · positive canary
The arm’s file loads
We put a unique marker in the local CLAUDE.md file. The model's output included this marker. This proves it read the correct instructions for the test.
04 · negative canary
Nothing private leaks
We checked for six secret markers from our internal environment. These included tool names, a private server name, and specific product words. None of them appeared in the output.
The models had no knowledge of our real product or its strategy. They had no other project instructions. We measured the failure and the fix in a totally clean environment. The logs are saved. You can re-run the test with validate-isolation.mjs.
New — tested against a published system
We tested the state-of-the-art fix. It never beat a one-line instruction.
The obvious objection to all of this is: “a real memory system would handle it.” So we built the leading published one as an arm. TierMem (Feb 2026) keeps a short summary and a complete raw log, and escalates to the raw log only when the summary looks insufficient. We gave it every advantage: the raw log is complete, and the summary openly states it is a snapshot from three months earlier.
We predicted it would fail. We were half wrong, and the real answer is more interesting.
A published two-tier memory architecture with a smart router never beat the one-line instruction “check the notes first,” and did worse on six of the ten models we tested, and better on none. On the Claude models it scored identically, trial for trial. The four ties are the three models where “check the notes” also fails, plus Opus where both saturate — there is no model on which the router is the better choice.
The reason is the interesting part: escalation is all-or-nothing. When the router escalated it read 6 notes and scored 6/6. When it didn't, it read 0 notes and scored 0/6. There is no middle. It never partly works — it either looks, or it trusts a summary that is complete, confident, and wrong. On GPT-5.4 that decision came out differently across identical runs (3 of 6), which is the worst thing a safety control can do: be unpredictable.
This is our point exactly. A sufficiency check asks “is this summary enough?” A stale summary is always enough. It answers every question. It just answers them wrong.
Does this survive outside a test we wrote ourselves?
We re-ran it on 272 cases written by other researchers
Every number above this point comes from one scenario we invented — and we invented it with the same model family we were testing, which is a real weakness. So we took STALE, an outside benchmark of 400 expert-written cases where a new fact quietly invalidates an old one, and rebuilt it in our setting: the stale value in the always-loaded summary, the current value in one dated note among 40 look-alike notes, and their question as the task. 272 of the 400 cases converted cleanly. The answer key was extracted by a different, weaker model and admitted only by a mechanical check, so it is not ours.
Read the first column, not the second. STALE's questions are deliberately indirect (“recommend somewhere to relax”), so a perfectly good answer often never names the value at all — our keyword scoring counts that as neither right nor wrong, which understates every method, the push most of all (23 of its 40 answers simply didn't mention the fact; none were wrong).
On cases we did not write: the trusting agent gave the outdated answer roughly two-thirds of the time, with zero tool calls on every single one. “Check the notes first” was no better than not asking (68% vs 65%) on the weaker model. The push gave an outdated answer zero times in 40.
We set out to prove our own feature works. It doesn't — at least not this way.
Our “graph” idea had a premise. We tested it on a real codebase and the premise was false.
The product idea we had not yet earned was this: in a big enough archive, ordinary search truncates — it returns the top handful of matches and quietly drops a related record two hops away — so a structured graph of “this replaced that” links should win. We built the test on a real project: every one of Wagtail's 223 versioned release notes, 2014 to 2026, pinned to an exact commit. That is 1,538 searchable chunks; asking for the top 10 sees 0.65% of it. Then 15 real deprecation chains, each hand-checked, where answering needs two different documents — the release that removed a feature, and the release one to two years earlier that deprecated it.
We also deliberately made our own opponent stronger than planned. The obvious baseline was the lab's existing similarity search, which is weak at exact identifiers — beating it would have been a rigged fight. So the baseline became BM25, a classic keyword ranker that is very good at rare identifiers, plus hybrid and recency-ranked variants.
Plain keyword search answered every single question. 15 of 15, with 27 of the 30 required documents landing in the top 5, and the worst-placed one at rank 29 out of 1,538. Nothing was out of reach. There was no buried record that only a graph could find — so we stopped, built no graph arm, and spent no money on models.
The lesson generalises, and it corrects how we had been thinking: the thing that makes search truncate is not how big the archive is — it is whether the question and the document share words. A unique identifier behaves like a hash key; you can grow the archive to a million documents and it still comes back first. “Bury it in thousands of notes” was simply the wrong mental model, and it was ours.
Two things we did to avoid fooling ourselves — and one that backfired usefully. Before publishing a negative result we checked we had not built a test where no effect was possible: we re-asked all 15 questions the way a developer who doesn't already know the identifier would ask. Everything inverts — keyword search collapses from 13 to 5, and at top-10 the best method finds only 7 of 15. So a real gap does exist, just not the one we claimed.
We also dropped one chain entirely: the corpus never announces that deprecation, so grading it would have scored every method on information none of them had.
And our first write-up claimed a tidy structural finding — that similarity and recency each find one hop and miss the other, so neither can win. Then we built the steelman version of the recency method we had just criticised, and it refuted us: blended properly it holds the older hop at 15/15 everywhere and becomes the best method in the grid. We deleted our own nicer finding.
A result that went against us
We checked whether real projects' context files actually go stale. Mostly, they don't.
Our whole premise is a context file that has fallen behind the project. That premise deserved testing outside our own fixture, so we took the 12 real repositories used by the AGENTS.md study ↗ — 10 of which ship a context file their own maintainers wrote — and mechanically checked every file path those documents mention against the repository as it stands today. A file path is a claim you can verify without opinion: either it exists, or it doesn't.
This is a negative result for the thing we most wanted to confirm, and we are reporting it as one. On the one dimension that can be checked mechanically, hand-written context files are well maintained.
Two honest qualifications, in both directions. File paths are the easiest thing to keep current, because they break loudly — prose claims (“the default is X”) rot silently and this check cannot see them, so 2.1% is a floor on a best case, not an estimate of overall staleness. And the more useful finding came from reading the files rather than counting: real hand-written context files are overwhelmingly instructions — “run the checks before you finish” — not claims about what is true. There is very little in them that can go stale the way our fixture does.
So we are narrowing our own claim: this failure applies to context that asserts facts — machine-generated repository summaries, memory files, auto-summaries, and our own push payload — and not to hand-written contributor guides. That also explains the AGENTS.md study's own split, which they left unexplained: generated summaries assert facts and made things slightly worse; hand-written guides assert practices and helped slightly.
The strongest rival — and where it got there first
Someone else published this “governed memory” idea before we did
TierMem is not the rival worth worrying about. CUPMem is — and it is only fair to say plainly that it published the core idea this project had been describing as its differentiator, first, with a working prototype and a number. It appears in the same paper as the STALE benchmark (May 2026).
CUPMem does three things. When new information arrives it labels every older fact keep / stale / replace / unknown. It then propagates that judgement to related facts the change implies. Finally it constrains what the agent may read: active facts ground the answer, stale ones are demoted to history, and unresolved ones block old defaults. That is close enough to the governance model here that we are not going to claim the idea.
Detection is close to solved. Application is not — and application is the whole job. CUPMem spots an outdated fact about nine times in ten, but applies the corrected fact in downstream work only a third to a half of the time. The task on this page is entirely the second kind: write the document, using the facts, correctly.
Where this study still sits apart: CUPMem governs what the agent reads. It runs when a question is asked and memory is searched. The failure measured here happens before that — the agent never searches, because the summary already looked like an answer. A rule about what may be read cannot fire when nothing is read. The two look complementary rather than competing, and we would rather say that than pretend we got there first.
So we built it as an arm and ran it against everything else. CUPMem's three parts are described well enough to rebuild: label every older fact when new information arrives, propagate that judgement, then restrict the agent to reading only what is still marked live. The store was built by a model reading the raw notes — not written by us. Hand-writing it would have made it perfect by construction, which proves nothing.
1. It rescues the weaker model outright — 0.0 to 15.5. Both the stale summary and “check the notes first” score zero on Haiku. Deciding what is current ahead of time, and handing the model the result, is the single biggest improvement any method other than the push has produced on a weak model. That is a real win and we are not going to bury it.
2. But on the strong model it lands BELOW doing nothing — 14.0 against a 16.0 floor. Left alone, Opus reads the whole log and works it out. CUPMem restricts it to a pre-built store that is itself only 13/20 — so the safety mechanism becomes a ceiling, and blocks the recovery reading the raw notes would have given.
3. Its ceiling is its store. Graded on its own the store scores 13/20; the finished work scores 13–16/20. It can never exceed what the summarising step captured. Deciding what matters before you know the question is lossy, and the restriction stops you recovering from it.
4. It breaks exactly where we predicted, before running it. Its four labels — keep, stale, replace, unknown — have no way to say “this applies to one customer tier only; the default is unchanged.” On that trap it scored 0 of 4. On the trap we thought would be the real test — a newer proposal that was rejected — it scored a clean 4 of 4.
What this is and is not. This is CUPMem's approach rebuilt under our conditions: one adjudication pass by a small model, on a product repository rather than the personal-assistant chat their benchmark uses. It is not their system on their benchmark, and their 68.0% there stands unchallenged — do not read our 14.0/15.5 as a refutation of it. The percentages in the table above this section are still theirs, on their task, and should not be compared to any /6 or /20 score on this page.
Caution on the percentages above: they are CUPMem's scores on their benchmark — personal-assistant memory in chat, not a code repository — and their evidence sits in the conversation the model can already see, where the evidence here sits in files it must choose to open. Do not compare these numbers to the /6 scores elsewhere on this page. Different task, different scoring, different underlying model. The comparison being drawn is structural, not numeric.
Independent confirmation from another lab
A different lab found the same thing, a different way
In June 2026, researchers at Snowflake published a paper, “Plans Don’t Persist,” on a different question: does an agent remember its own plan during a long task? The Snowflake team also used a different method, looking inside the model’s hidden state rather than at its behavior. It arrived at the same core fact: an agent’s important information lives in the text of its context window, not in the model’s memory. Once that text is dropped or compressed away, the information is gone. The paper’s phrase for it: “context management is load-bearing.”
Two teams, two methods — the Snowflake paper probes what the model holds internally, this study measures what the model actually does — and one shared lesson: the context an agent works from is not something it “knows,” it is something that has to be kept in front of it. That is the whole reason keeping it current and selective matters. (The Snowflake paper adds an honest caution worth repeating: simply re-showing old context does not automatically fix things. That is why the approach tested here surfaces the current facts for the task at hand, rather than any past text.)
Three more papers, and where this result sits among them
This area got crowded fast in 2026. Here is an honest map of the neighbouring work — including the places where other researchers got there first.
Put together: earlier work has shown that agents ignore evidence they have already retrieved, and that project context files quietly fail to help. The contribution here is the step before both of those — a confident summary stops the lookup from ever happening — along with the finding that this is why context files do not pay off. The adjudication idea is not claimed here; it belongs to CUPMem.
How to understand these results — and what they don't show
- We know “perfect maintenance” isn't real. We included it on purpose. No one updates their notes perfectly all the time. Panel 1 just shows that the note format isn't the problem. Panel 2 shows what happens in the real world.
- Correction (21 Jul 2026): we previously said the push “does not work automatically in some setups.” That was wrong, and it was our mistake. We had recorded that the hook which injects current facts could not fire in automated (headless) runs. We finally tested it directly, and it fires fine — we confirmed it with a secret phrase that existed nowhere except the hook’s output, which the model then repeated back. What actually happened is that our own test harness deliberately strips that hook to keep the machine’s real setup out of the experiment, and we wrote that down as if the platform couldn’t do it. An isolation choice got mistaken for a limitation, and it went unchallenged for the length of this study. The still-true part: in the runs on this page the push is supplied by hand rather than by the live hook, so these numbers measure the mechanism, not our shipping integration. We are re-running them properly.
- New: we are moving off our own hand-built test case. Every number above comes from one scenario we wrote ourselves — and we wrote it with the same model we were testing, which is a real weakness. We have now ported 272 scenarios from STALE, an outside benchmark written by other researchers, into this on-disk setting. First results reproduce the finding on someone else’s test cases: the trusting agent shipped the stale answer 5 times out of 6, and the push shipped zero stale answers. Full numbers when the larger run finishes.
- Our “graph” feature was not the reason for the correct answers here. In other tests, a good model can figure out how notes relate to each other from the text alone. The graph was not needed. The graph's real strength is finding all related notes in a very large project. This small test cannot show that, so we are not claiming it here.
- We tested ten models from three companies. This is not a final proof that it works everywhere. The failure and the fix held for Anthropic, Google, and OpenAI models, across three successive OpenAI generations (GPT-5.4, 5.5, 5.6) — but every one is a closed commercial model. To prove this is a universal problem, we still need to test open-source models. We also need to run more trials. Gemini's results in particular varied a lot. So, its numbers show a trend, but are not exact. The most reliable results are for floor (0/6) and push (5.0–6.0). We only tested RAG on the Claude models. We used verify to represent search-based methods on Gemini and OpenAI.
- New (21 Jul 2026): the 6-fact test has a ceiling, and a harder one shows what that ceiling hides. Every method that looks at the notes at all scores 6/6 here, so the top half of these tables cannot be ranked — that tie is the test running out of room, not the methods being equal. So we built a harder version: 20 facts across five kinds of trap (plain recency, a newer decision that was rejected, an exception that applies to only one tier, a multi-step calculation, and a near-duplicate draft), hidden in 100 neutrally-named notes. It ranks cleanly, with no ceiling — and it revises one reading above. “Check the notes first” is far less stable than a tidy 6/6 suggests: on identical inputs it scored 19/20 on one run and 10/20 on the next. The push scored 20/20 both times. But the harder test cuts against us too, and that matters more. On it, the stale-summary failure stops reproducing on the strongest model: faced with 20 facts, Opus treats the summary as obviously incomplete and simply reads the whole corpus (100 notes in 3 tool calls), scoring 16/20 from the stale summary alone. On Haiku it still fails completely (0/20). We think that sharpens the claim rather than shrinking it: this failure bites hardest when the stale summary looks like it already covers the job — which is the normal case for a project summary, and the case Panel 1 tests. The two fixtures measure different things — whether the agent looks, versus whether it can reason over what it finds — and we do not mix their numbers.
- These results are early and internal. We ran 3 trials for each test on the 6-fact fixture and 2 on the 20-fact one. The important signals are the average scores and when the model made 0 tool calls. We are not reporting statistical significance yet, and the run counts above are too low for it — the instability we found in “check the notes first” is exactly the reason we are raising them.
Reproducibility — read this before trusting any number above
What you can check yourself today, and what you cannot
We would rather state this plainly than let the commands below imply more than they deliver.
The harness is now public: github.com/Meetless/stale-context-bench — MIT, fabricated fixtures, every runner and grader included.
So the honest summary is: the phenomenon is the reproducible part; the product claim is not yet. The failure, and the ranking of the mitigations against it, rest on fixtures anyone could rebuild. The rows labelled “mla push” are a measurement of an idea, not a benchmark of a shipping integration — and until the harness is public and that arm runs the real product end to end, they should not be cited as evidence about the product. Both of those are being fixed; neither is done.
git clone https://github.com/Meetless/stale-context-bench
cd stale-context-bench
node validate-isolation.mjs --model claude-opus-4-8 # prove the clean room
node matrix.mjs --trials 3 --conc 3 # all arms x {opus, haiku}
OPENAI_API_KEY=... node openai-matrix.mjs --trials 3 # cross-vendor
node run.mjs --arm hard-push --model claude-opus-4-8 --trials 2 # 20-fact graded fixture
node cupmem-adjudicate.mjs --arm arms/hard-cupmem # rebuild the CUPMem store, then run it