The setup, the four hops that all have to fire, the negatives that are supposed to hurt, and the two bugs we found in our own benchmark before they became findings.
Two engineers, each with an agent, working at the same time. One decides the production rollback window is 43 minutes. The other, an hour later in a different terminal, decides it is 16. Nobody is in the wrong; nobody is told. The number that ships is whichever one gets written down last.
Every other benchmark on this site pits governed memory against a vanilla agent. This one cannot. A vanilla agent is not weak at this task — it is structurally incapable of it. Detecting that another session decided something incompatible requires (a) both sessions writing into a shared memory and (b) something reading across them. An agent with a CLAUDE.md has neither: the other session's decision was never written anywhere it can see. Handing it a losing score would be theatre.
So the question here is absolute, not comparative: does the lane work, how fast, and how often does it cry wolf.
Detection is not a function call. It crosses two services and a job queue, entirely asynchronously, after the agent's turn has already ended:
session A turn closes -> Stop hook -> flush -> control: AGENT_RUN_FINALIZED -> worker -> intel: EXTRACT_TURN -> normalized claims -> INDEX_CLAIM (now retrievable by other sessions) session B turn closes -> same chain -> the claim scanner searches for neighbours -> finds A's claim -> contradiction classifier -> confidence gate -> CONTRADICTS relation -> CoordinationCase opened
Every one of those hops is silent when it fails. Capture is deliberately fail-soft — an agent should never be interrupted because a governance sidecar had a bad day. Which is exactly what makes this lane treacherous to benchmark: if hop 1 quietly does nothing, the run still completes, the detector never fires, and the naive scoreboard reads 0% recall — the detector is broken.
So the harness proves each hop landed, against the database, before it will score anything: the agent run reaches completed; EXTRACT_TURN completes; INDEX_CLAIM completes. A pair whose chain does not land is scored invalid and excluded — never counted as a miss. We refuse to blame the detector for a delivery failure upstream of it. (In the published run: 0 invalid pairs.)
Version one of this runner would have reported a confident 0% recall and blamed the product. Its sessions never registered with the backend at all — for two independent reasons, both of them silent by design.
First, each pair created its workspace under a throwaway owner, so the agent's own identity was not a member of it. Every capture request came back 403 and capture paused itself, exactly as it is designed to. Second, the sessions ran in a bare temporary directory, and the finalize step refuses to post without git evidence — so no run was ever finalized, and nothing downstream of it ran.
The result was a benchmark that ran to completion, produced a plausible number, and measured the harness. We found it only because we gated the hops and watched hop 1 report zero agent runs in the database.
The first version of the delivery check spawned a fresh single-turn session and asked it what the rollback window was. It answered, honestly, that it had searched governed memory and found nothing — which reads exactly like a damning retrieval failure, and we very nearly published it as one.
It was not. Conflict warnings reach an agent through a session-scoped cache that is written when a turn ends and read when the next turn begins. A brand-new single-turn session cannot receive one: its cache is first written by its own closing flush, one turn too late. That is a property of the transport, not a failure of detection, and scoring it as one would have been a lie. The delivery check now resumes session B and uses the product's own machinery end to end.
This product has measured a ~86% false-positive rate on its own live conflict queue. Against that, a benchmark fed nothing but genuine contradictions would score close to 100% recall and be worthless — it would measure how eagerly the detector fires and call that success.
So most of the pairs here are decisions that look like conflicts and are not. They are the whole point, and they are graded exactly as harshly: a false alarm costs what a miss costs. An engineer who is warned about nothing learns to ignore the warning.
| Archetype | Session A | Session B | Why it is compatible | Alarms |
|---|---|---|---|---|
| Different setting the easy one |
rollback window is 18 minutes | staging deploy timeout is 277 seconds | different entity, different unit — a neighbour in embedding space, nothing more | 0/5 |
| Unit restatement adversarial |
rollback window is 18 minutes | …restated in the units our alerting uses: 1080 seconds | the same value. Nothing changed at all. A numeric differ walks straight into it | 0/3 |
| Scope carve-out adversarial |
rollback window is 31 minutes by default, all regions | in the EU specifically it is 43 minutes — a regional carve-out on top of the default | same setting, different numbers, both true at once | 1/3 |
The scope carve-out is the one that matters, and it is the one that fires. It is precisely the shape a "same property + different number ⇒ contradiction" heuristic cannot survive, and precisely the shape of the false positives on the live queue. We report it rather than dropping it.
Every rollback window, timeout and carve-out is a random integer generated at run time and written into the workspace through the product's real ingest path. The values exist in the database and in the launcher's memory, and nowhere else — not on disk, not in the repo, not in a fixture file. A model cannot have seen them, and a re-run cannot be tuned against them. Each pair also gets a fresh workspace, so a case left open by an earlier pair can never be counted as this pair's detection.
A CoordinationCase with two distinct session sides, or a CONTRADICTS relation, observed by polling for up to 5 minutes. Polled, never sampled once: the chain crosses a queue, and a single sample would measure the queue rather than the product.
Detection that stops at the database is not a product. So after a case opens, session B is resumed and asked a neutral question — "I'm about to write the rollback runbook. What is the rollback window, and is there anything I should know?" — with no mention of any conflict. It scores as delivered only if the agent raises the contradiction itself, or surfaces both decided values unprompted. On the negatives, the same question is asked to check the agent is not warned about a conflict that does not exist.
tools/dev-stack.sh up # control + worker + intel, health-gated tools/bench-mla/preflight.sh # refuses to run against a lying stack node tools/bench-mla/b5-parallel-sessions.mjs --probe # one pair, prints every hop node tools/bench-mla/b5-parallel-sessions.mjs --pairs 5 # 5 positive + 5 negative node tools/bench-mla/b5-parallel-sessions.mjs --hard --pairs 3 # 6 adversarial negatives
The preflight is not decoration. It refuses to benchmark when the worker's queue is backing up, when intel is serving from the wrong virtualenv, or when a stale process holds the job-poller lock — three faults that each cost us hours and every one of which still answered its health check with a 200.