CLAUDE.md (and AGENTS.md / .cursorrules)

This is a Markdown file the agent always reads. It loads at the start of every session. It's the default way to give an agent context. But what if the file's facts go out of date? That's the failure this benchmark tests.

Baseline · default Always-loaded every session Advisory, not enforced Our benchmark: 0/6, zero tool calls

What it is

Anthropic says CLAUDE.md is “a special file that Claude reads at the start of every conversation.” The file system is hierarchical. The agent looks for this file in parent directories. It combines every CLAUDE.md it finds. It also loads files in subdirectories when needed. Other tools have similar files, like Cursor’s .cursorrules and AGENTS.md.

One detail explains most of its problems. The file is advice, not a command. Anthropic is clear about this. The file is just context for the model. It is not an enforced rule. They say “there’s no guarantee of strict compliance” (Anthropic docs).

Setup

Run /init. Claude will analyze your code. It looks at your build system, tests, and conventions. Then it writes a starter CLAUDE.md file for you.
Edit the file by hand. Then commit it to git so your team can share it. Add things the agent can't guess. This includes special commands, code style, or architecture notes. Put your personal notes in CLAUDE.local.md, which should be gitignored.
You can add nested CLAUDE.md files in subdirectories. These files contain rules for specific parts of your project. They only load when the agent works in that directory. Put preferences for all your projects in ~/.claude/CLAUDE.md.
Keep the file up to date. Use /memory to edit it. Use /context to see what loaded. Use # to add a rule during a session. Anthropic says to treat it “like code: review it when things go wrong, prune it regularly.”

Sample directory structure

~/.claude/CLAUDE.md              # user-global: personal prefs, all projects

your-project/
├─ CLAUDE.md                     # project root: shared, committed to git
│                                #   commands, code style, architecture, rules
├─ CLAUDE.local.md               # personal project notes (gitignored)
├─ .claude/rules/                # optional split-out, path-scoped rules
│  ├─ testing.md
│  └─ api-design.md
└─ src/billing/
   └─ CLAUDE.md                  # nested: loads on demand under src/billing/

Files load from general to specific. The order is: managed policy, then user-global, then project, then local. Parent directory files load at the start. Subdirectory files load only when needed. Note that splitting files helps you stay organized. It does not reduce token costs. Everything you reference still gets loaded.

Pros & cons

Strengths

Simple and requires no setup. It's just one Markdown file. You don't need a database or other complex tools. It has no required format.
Always loaded automatically. The file loads at the start of each session. You don't have to do anything. A file at the project root even stays loaded after /compact.
Shared with your team via git. You can commit the file to your repository. This way, project rules travel with the code. Every teammate gets the same instructions.
Flexible and layered. You can nest files in different directories. You can also set rules for a user, a project, or a whole organization. This lets you layer general rules with specific ones.

Weaknesses

The model ignores it sometimes. The file is just advice, not a command. The model may not follow the instructions. This gets worse as the file gets longer.
Longer files work less well. The more rules you add, the less the model pays attention.
The information gets out of date. A summary might be correct when you write it. But it can become silently wrong later. This is the exact problem our benchmark tests.
It costs tokens on every turn. The file is loaded into context at the start of every session. This uses up tokens before you even type.
It forgets rules in the middle. The model often follows rules at the start and end of a session. But it may ignore them during the main task.
It requires constant updates. Someone has to manually edit the file to keep it current.

Common user feedback

Anthropic’s own documents admit the two main problems. That's why we use this method as our baseline. It's a real-world problem, not a made-up one.

Long files cause the model to ignore rules. Anthropic agrees. Their own guide says: “Bloated CLAUDE.md files cause Claude to ignore your actual instructions!” and “important rules get lost in the noise” (Anthropic best practices). One developer used a 200-line file and found: “Claude no longer knew which rules mattered most” (XDA).
Sometimes, the model just ignores the file. One GitHub bug report is titled “Claude Code continually ignores CLAUDE.MD file” (#34197). A user on Hacker News said: “Claude Code just stops following a lot of the directions specified in the file” (HN).
The file gets stale over time. One article notes: “Most teams set up CLAUDE.md once and never touch it again… Three months later, Claude is ignoring parts of it, following instructions that no longer apply.” (aicodex). Anthropic also warns against putting “information that changes frequently” in the file.
It uses tokens every session. Anthropic confirms this. They say the file is “loaded into the context window at the start of every session, consuming tokens” (docs). One developer explained: “A 5,000-token CLAUDE.md costs 5,000 tokens before you’ve typed a word. Every turn.” (buildtolaunch).
It follows rules at the start and end, but not in the middle. One user said: “Claude would adhere… at the beginning and end of the conversation, but was likely to ignore during the middle where the real work is being done.” (HN). Other tools have the same problem. A Cursor user reported “AGENTS.md is ignored although documentation states it should respect it.” (Cursor forum).

How it did in our benchmark

This method is our baseline. We created a CLAUDE.md file with six product facts. All six facts were later changed. The score shows how many facts the agent got right out of 6.

metric	every model (all 10, 3 vendors)
facts current	0 / 6
source notes read	0
variance across models	none — universal

The agent could have found the right answers. They were in the notes/ folder. But it didn't look for them. Why? The summary in CLAUDE.md gave it a confident, but wrong, answer. So the agent never checked for new information. It made zero tool calls and gave stale answers. This happened with every model we tested. A summarizing CLAUDE.md is a trap. The more you trust it, the worse the problem gets. See the full results →

This page is part of the methods reference for the stale-context benchmark. We flag vendor sources like Anthropic's docs. Every quote links to its original source.