Nobody told either of them. Fifteen seconds after the second one decides, it finds out — and refuses to write the number into the runbook.
This is the one benchmark on this site with no baseline arm, because there is nothing to compare against. A vanilla agent cannot lose this test — it cannot take it. Detecting that another session, running right now in another terminal, just decided something incompatible requires a shared memory that both sessions write to and something that reads across them. An agent with a CLAUDE.md has neither.
So the question is not "does it beat vanilla". It is does it work, and does it cry wolf.
Session A decides. Session B, running in parallel, decides something incompatible. Every hop below is asynchronous, and every one of them is silent when it fails. So the benchmark gates on each one: a pair whose chain does not land is scored invalid, never a miss. We refuse to blame the detector for a delivery failure upstream of it.
Two sessions each decide a production rollback window — 43 minutes and 16 minutes. Both cannot hold. On its next turn, session B is asked a neutral question: what is the rollback window, and is there anything I should know? It was never told a conflict exists.
The rollback window was set to 16 minutes in this session, but that decision is currently under a coordination hold because it conflicts with an in-flight change in another session. Don't commit the 16-minute figure to the runbook yet. The agent, unprompted, on its next turn
A detector that only ever sees real conflicts will score 100% and tell you nothing. So most of this benchmark is decisions that look like conflicts and are not.
| Negative archetype | False alarms |
|---|---|
| Different setting entirelyrollback window (min) vs staging deploy timeout (sec) | 0/5 |
| Unit restatement"18 minutes" vs "1080 seconds" — the same value | 0/3 |
| Scope carve-outglobal default 31 min + "in the EU it is 43 min" — both true | 0/2 |
We first published 83% precision. One negative pair opened a contradiction case for two decisions that are both true at once — and warned the next agent about it. We went looking for the flaw in the detector and did not find one.
The session had said "the rollback window is 31 minutes by default". What our own claim normalizer stored was "the rollback window is 31 minutes in all regions". It had deleted the word "default", turning a defaultable rule into a universal one — and "31 minutes in ALL regions" genuinely is refuted by "in the EU it is 43".
The detector was right about the text it was handed. The text had been falsified upstream, by us. Of three identically-shaped pairs in that run, the two whose normalization kept the word coexisted correctly, and the one that lost it raised a false alarm.
| Metric | Published | After the fix |
|---|---|---|
| Recall | 5/5 | 5/5 |
| Delivered to the next turn | 5/5 | 5/5 |
| False alarms | 1/11 | 0/10 |
| Precision | 83% | 100% |
The fix is a deterministic guard in the claim validator: if a quote carries a defeasibility marker and its restatement carries none, the claim was strengthened and is rejected. We did not fix it with a prompt — the bug was non-deterministic and would not reproduce on demand, so a prompt change could never have been honestly verified. Recall did not move, which is the number we actually needed to check before claiming the fix was free.
We are leaving the 83% on the record. A benchmark that finds a real bug in the product, gets it fixed, and then measures the fix is worth more than one that scored well on the first run.