A Governance-Catch Census: What Our Agent Harness Caught in One Day — and What Got Through First
July 27, 2026 by NXTG.AI
Publisher: NXTG.AI · Window: 2026-07-01, a single operating day · System: a live production multi-agent system — long-running agent lanes across two machines doing real engineering and operations work.
Why publish this
Vendors publish what their monitoring can do. We wanted to know who publishes what their monitoring actually caught, with the misses included. In our non-exhaustive surveyed set — commercial vendors, safety-evaluation organisations, frontier labs and telemetry aggregators, probed on 2026-07-27 by a method that is disclosed internally but not independently re-runnable, and which left four classes unprobed — we found no party publishing an operational wrong-claim interception census with a public counting rule and public per-item evidence. This is not evidence of universal absence. That set included us, which is why this document exists.
The number here is not flattering, and that is the point: at least 9 caught before they shipped, and at least one that reached a running server before it was caught.
1. The counting rule
Every item is assigned to exactly one of four classes. The classes are reported separately and are never summed into a single "wrong claims" total — blending them is the error that produced our earlier, retired figure.
| Class | Definition |
|---|---|
| caught-wrong | A confirmed-wrong state claim caught before it shipped. Includes near-misses caught pre-ship. |
| shipped-wrong | A wrong state that reached a running system before it was caught. Note the precision: not uncaught — caught, but only after reaching runtime. Counted against us. |
| process-error | A real behavioural error that is not a falsifiable state claim (e.g. a runaway loop). Excluded from the state-claim numerator. |
| verified-TRUE | A claim independently checked and found correct. A verification event, not an error — must never appear in a wrong-claims numerator. |
Result for 2026-07-01 — observed classified enumeration; both wrong-claim counts are floors, see §2: caught-wrong ≥ 9 · shipped-wrong ≥ 1 · process-error = 1 · verified-TRUE = 1
2. What these numbers do and do NOT support
They do not support a catch rate. An earlier draft of this document claimed the rate was "≤90%, a ceiling not a score." That inference is invalid and we are retracting it before publication rather than after. caught-wrong ≥ 9 and shipped-wrong ≥ 1 are floors that move the quotient in opposite directions — additional caught items raise it, additional shipped items lower it — so the pair constrains nothing about the true rate. The error was easy to miss precisely because it was dressed as conservatism.
What is actually true:
- The observed classified ratio is 9/10 within this retrospective enumeration.
- The population catch rate is unidentified from these data. There is no valid one-sided bound, in either direction.
- Both counts are floors over an enumeration known to be incomplete: unlogged catches may exist and are unobservable to this ledger-derived enumeration, and additional shipped-wrong is unknown pending audit.
Selection is not preregistered. This window was chosen because it had already been written up as an internal postmortem — it is retrospective and outcome-salient, not a randomly or prospectively selected day. That may bias the window in either direction, and no claim here should be read as a representative sample.
3. Per-item record — all 12 items, including the one that got through
| # | What was claimed / done | Class | How it was caught |
|---|---|---|---|
| 1 | "12 stuck message-queue rows should be expired" | caught-wrong | Probe at the authority source showed the rows were already-processed inbound messages, not orphans |
| 2 | "a scratch directory is not ignored — live bomb" | caught-wrong | An ignore-check against canonical origin settled it in one command; the alarm was a stale local view |
| 3 | A server fix was silently reverted (a shell metacharacter executed inside a payload) | shipped-wrong | Caught — but only AFTER the reverted state existed in the running server. A second layer of defence masked user impact. This is the one that got through. |
| 4 | An agent self-reported a commit as pushed | verified-TRUE | Fetch-then-judge confirmed it was pushed — a verification event, not an error |
| 5 | "the sync wedge may have eased" | caught-wrong | Lock-held ticks were bailing before commit — masked, not cleared |
| 6 | "the fix is to add the file to the path-pattern list" | caught-wrong (near-miss) | Wrong mechanism — it was the trigger set, not the pattern list; caught before it shipped |
| 7 | A watcher declared deploy-truth green | caught-wrong | It had matched a historical commit — no time filter on the probe |
| 8 | "the journal shows no device lines" (hardware fallback suspected) | caught-wrong | Faulty search pattern; a journal re-read refuted it |
| 9 | "cross-instance query is built" | caught-wrong | Roadmap, not built — self-caught mid-turn; the claim was factually false |
| 10 | "the cached-file removal fixed the dirty files" | caught-wrong | A skip-worktree bit had silently defeated it |
| 11 | Relay-loop inflation — every acknowledgement triggered a full cycle | process-error | Real and wrong, but a behaviour error, not a state claim — excluded from the numerator. Weakest row in this table: its only evidence is the retrospective postmortem plus an escalation record; no independent instrument survives and it is NOT independently certified. |
| 12 | "the two certs contradict each other on recall" | caught-wrong | The contradiction claim was itself false — an instrument-reconciliation error |
Read row 3 first. A census that lists only successes is marketing. Including the miss reduces one cherry-picking concern; it does not validate any other row. Read row 11 second — it is the row we are least able to evidence, and we are telling you so rather than letting it pad the table.
4. Evidence status — stated honestly, not claimed green
This table is a redacted per-item summary, not a set of independently checkable receipts. It provides no public evidence URL, content hash, or third-party attestation for any individual row. A reader can check our counting rule and our reasoning; they cannot yet independently verify any single row against source evidence.
We are not claiming otherwise. Publishing per-row evidence with stable public identifiers is the next build, and until it exists this document is a disclosed methodology plus a summary — which is more than we found in our non-exhaustive surveyed set, and less than proof.
5. Method, and how to challenge it
- Unit: one state claim or action per row, assigned to exactly one class.
- Independence: the author of the original blended figure did not grade this partition. It was re-derived row-by-row from the operating record by a separate evidence pass, then adversarially reviewed by a different vendor's model on a separate context — which is what caught the invalid rate inference in §2 and the missing caveat on row 11, before publication.
- The structural limit — a ledger-derived catch rate is ascertainment-biased. You can only count the misses you eventually noticed. This does not mean such a rate always flatters in any given sample; it means the instrument cannot distinguish perfect catching from undetected misses. Because the ledger omits undetected misses by construction, a 100% observed ratio cannot distinguish perfect catching from unseen misses and is not evidence that no misses occurred — from any party, including any future figure from us.
- What would change these numbers: any additional shipped-wrong found in the pending audit. We will publish that too.
- Other limits: single-day window; single-organisation system; retrospective and outcome-salient selection; enumeration incomplete by construction; not a controlled experiment and no comparison arm. This is practice observed, not a benchmark.
6. What this is not
It is not a claim that our catch rate beats anyone's — we found no comparable published operational census in our non-exhaustive surveyed set, and this document does not establish a rate at all. Its value is that the counting rule is disclosed and the miss is included. If you publish a competing census, we will link it.
Redaction note: internal host names, machine identifiers, absolute paths and internal repository identifiers are omitted; every row preserves the substance of the claim and the nature of the check that settled it.