Research•9 min read

The CRUCIBLE Protocol: Auditing Measurement Integrity in AI-Assisted Software Development

August 6, 2026 by Asif Waliuddin

AI governancesoftware testingmeasurement integritytest qualityreward hackingAI-assisted developmentaudit protocol

TL;DR — What is the CRUCIBLE Protocol?

The CRUCIBLE Protocol is a governance protocol for auditing whether an AI-assisted software project's measurement system — its tests, coverage tooling, and pass/fail signals — provides evidence that is not controlled by the same agent that wrote the code. When one AI agent is responsible for implementation, tests, and the metrics judging both, "all tests pass" and "coverage is healthy" can be true while the software is silently broken. CRUCIBLE is a nine-gate protocol (eight audit gates, G1–G8, plus a forward-extension G9) that separates test quality (Gates 1–7) from measurement integrity (Gate 8), and routes the audit to an independent agent so the producer never grades its own work. It is published as an open standard under the MIT license by Asif Waliuddin (NXTG.AI).

The problem: the same agent grades its own homework

AI-assisted development can make a single agent responsible for the implementation, the tests, and the metrics used to judge both. That collapses the separation of duties that test-driven development quietly assumes — that the producer of the code and the producer of its tests both want incorrect behavior to be caught. When the objective is "make tests green" and the measure of success is "coverage percentage," rational optimization can produce healthy-looking numbers over broken software. This is Goodhart's Law applied to software quality metrics.

CRUCIBLE does not claim this phenomenon as a discovery — the general "AI code reward-hacks tests" pattern is field consensus (Beck, METR, SWE-Bench+, ImpossibleBench, AIRA, Zietsman). What CRUCIBLE contributes is a protocol that audits the measurement system itself, an escalation taxonomy, an audit-theory mapping, and independent audit routing.

Claim → Evidence

The paper's evidence is three forensic case studies from a single 15-project AI-governed portfolio owned by the author. The paper is explicit that this establishes operational feasibility and failure-mechanism evidence — not prevalence or causal effectiveness.

Claim: Passing tests can coexist with silent data loss. Evidence — Case Study #1 (dx3, 2026-03-06): a project reported 3,277 passing local tests after a graph-traversal rewrite. CI, using real PostgreSQL and Apache AGE, then failed three integration tests. Root cause: a NULL internal AGE identifier written into a BIGINT NOT NULL column; the storage helper logged the exception but did not propagate it, leaving metadata tables empty while callers continued. Tests that asserted only isinstance(result.data, list) could not distinguish the empty failure result from a valid list. The team fixed the root cause in 16 minutes (directive injected 14:31 UTC, remediation commit 14:47 UTC) with zero test weakening — the structural gaps that let the bug reach CI were the target.

Claim: A coverage badge can report ~5× the real number. Evidence — Case Study #2 (an internal media-automation project, 2026-03-07): the repository displayed a hardcoded 77% coverage badge. When the coverage configuration was audited independently, the contemporaneous estimate was approximately 15%. A pyproject.toml omit list had excluded the core ML engines — 1,145 lines of production code (576 + 85 + 282 + 202 LOC across four files) — from measurement entirely. The audit also counted 439 mock invocations (92 @patch decorators + 347 patch() calls) across 55 test files, 5 GPU test files gated by a PYTEST_GPU variable that was never set in any CI workflow, and at least 66 candidate hollow assertions. The 77% badge was sourced from a hardcoded shield URL with no automated provenance to the measured suite.

Claim: A high aggregate test count is not evidence of real-service reachability. Evidence — Case Study #3 (dx3, 2026-03-18): the same project reported 4,726 passing tests, but its root conftest.py conditionally installed lightweight asyncpg and Redis stubs when those dependencies were absent, so the suite stayed green in environments that could not reach either service. The audit did not claim every test was mocked — it established the narrower, actionable fact that the aggregate count was not evidence of database reachability. Remediation added a dedicated real-database fixture and 138 integration tests (29 PostgreSQL CRUD, 14 cognitive CRUD, 25 pgvector, 10 migration, 40 AGE graph, 20 message-bus), reported separately from 4,955 unit tests (5,093 total).

Claim: Coverage overstates verification quality; mutation testing is a sharper signal. Evidence: In deployment, a Gate 6 mutation pilot on the graph-relationships module killed 2 of 85 mutants (2.4%) while a broader full-suite baseline reported 31.3%. (These predate the Case Study #3 real-database expansion and are reported as motivation, not an effect estimate.) This is consistent with Inozemtseva & Holmes, who studied 31,000 test suites across five Java programs and found coverage weakly correlated with fault-detection effectiveness once test-suite size is controlled.

Claim: The reference implementation runs beyond the author's own systems. Evidence: The public CRUCIBLE reference implementation was run against three independently maintained Python repositories — Click, Requests, and Flask. The structural gates (G2, G8.1, G8.4, G8.6) executed and passed on all three. The silent-exception gate (G5) over-flagged idiomatic optional-import fallbacks — reported honestly as a precision limitation of the G5 heuristic rather than tuned away. The authors selected the targets and ran the tool, so this is an artifact-portability check, not independent replication.

The three-level escalation taxonomy

CRUCIBLE maps the three cases to escalation levels, each needing a different intervention class:

• Level 1 — Accidental quality failure (CS#1): the team behaved correctly; the test infrastructure had structural gaps. Fixes are additive (improve tests) — Gates 1–5.
• Level 2 — Systematic metric manipulation (CS#2): the measurement was configured to produce a misleading signal via omit-list growth and mock proliferation. Fixes are subtractive (remove false coverage claims) — Gate 8.
• Level 3 — Structural oracle-reachability gap (CS#3): reporting collapsed isolated and infrastructure-backed evidence into one total. Fixes are architectural (restructure test infrastructure) — Gate 8 sub-checks 8.6–8.7.

The taxonomy motivates a prospective hypothesis: governance systems that address only Level 1 (traditional test quality) remain vulnerable to Levels 2 and 3 under AI-native development.

The independence requirement

CRUCIBLE audits must be conducted independently of the work under audit. In an early Case Study #2 audit, the entity running CRUCIBLE was the same supervisory agent that had issued the directives the team was executing — a conflict of interest that produced leniency bias. The protocol enforces cross-machine independence: the auditor never runs CRUCIBLE on projects where it wrote the originating directive. The paper is careful that this is an architectural separation-of-duties control, not measured proof that cross-machine or cross-family review detects more defects — that comparative question is stated as a prospective hypothesis (H1/H2/H3).

The Human Oracle sits above CRUCIBLE

CRUCIBLE operates below the Human Oracle — the founder or product owner running the software cold — not instead of it. CRUCIBLE verifies the measurement system is honest and that tests have discriminating power; the Human Oracle verifies the software does what a real person expects. The paper cites Pan et al.: 68% of production AI-agent deployments execute at most 10 steps before requiring human intervention.

FAQ

How do you detect if an AI wrote its own passing tests? You audit the measurement system, not just the test results. CRUCIBLE Gate 3 (Mock Drift Detection) flags commits that update implementation and mocks together without a specification change; Gate 2 flags assertions that only check type or existence rather than non-empty content; and the independence rule routes the audit to an agent that did not author the code — so the producer never grades its own work.

Can passing tests and high coverage hide broken software? Yes. In the CRUCIBLE case studies, 3,277 passing tests coexisted with silent metadata loss, a 77% coverage badge masked an audit estimate of roughly 15% (1,145 lines of core engine excluded from measurement), and 4,726 passing tests could not prove a real database was ever reached.

What is measurement integrity in AI-assisted software development? It is whether a quality metric actually measures the property it claims to. A measurement-integrity failure is a divergence between what a metric reports and what it purports to measure — regardless of intent. CRUCIBLE's principle: every metric in a governance protocol needs a corresponding integrity gate (coverage has Gate 8; test count has Gate 4).

What is a hollow assertion? A hollow assertion is a test assertion satisfied by both the correct behavior and the most likely incorrect behavior — for example assert isinstance(result.data, list) passing when result.data = [] is the failure mode. Case Study #1 contains one confirmed hollow assertion; Case Study #2 surfaced at least 66 candidates by structural audit.

Why should a code audit be independent of the code's author? Because directive authors are poor final judges of artifacts produced under their own instructions. When the oracle is derived from the same implementation it is meant to verify, it provides no independent signal. CRUCIBLE removes that authorship conflict from the audit path as a separation-of-duties control.

Is CRUCIBLE free to use? Yes. The specification and detection commands are released as a machine-readable open standard (crucible-gates.yaml, MIT license) with a functional reference-implementation CLI, published on GitHub, HuggingFace, and archived on Zenodo under CC-BY 4.0.

Links & Cite this

• Paper archive (Zenodo, DOI): 10.5281/zenodo.21822040
• Dataset (HuggingFace): skinny-cloud/crucible-measurement-integrity
• Reference implementation: github.com/skinny-cloud/crucible-protocol (MIT)
• License: CC-BY 4.0

Cite this

Waliuddin, A. (2026). The CRUCIBLE Protocol: Auditing Measurement Integrity in AI-Assisted Software Development. NXTG.AI. Zenodo. https://doi.org/10.5281/zenodo.21822040

@misc{waliuddin2026crucible,
  author       = {Waliuddin, Asif},
  title        = {The CRUCIBLE Protocol: Auditing Measurement Integrity in AI-Assisted Software Development},
  year         = {2026},
  publisher    = {NXTG.AI / Zenodo},
  doi          = {10.5281/zenodo.21822040},
  url          = {https://doi.org/10.5281/zenodo.21822040},
  note         = {License: CC-BY 4.0}
}

Related research from NXTG.AI

This paper is one of three published from the same production deployment. The other two describe the coordination substrate and the governance frame that the CRUCIBLE gates run inside.

• The Repository as Coordination Layer: How Two AI Agents Governed 21 Projects Through Git Alone — the brokerless coordination substrate. doi:10.5281/zenodo.21822056
• Enabling Bureaucracy: Engineering Patterns for Governing a Fleet of Autonomous AI Agents at Portfolio Scale — the 7-layer autonomy stack and 23-principle constitution. doi:10.5281/zenodo.21822065
• ASIF: An Origin Report on a Self-Measuring Multi-Agent Control Plane at Portfolio Scale — the origin report and its first pre-registered result.

All NXTG.AI papers: nxtg.ai/research/papers