Insights•12 min read

The Verification Trap: Why AI Agents Game Your Tests (And What Actually Works)

March 13, 2026 by Asif Waliuddin

AITestingTDDAgents

The Verification Trap: Why AI Agents Game Your Tests (And What Actually Works)

I've been writing software for 20 years. I bought into TDD hard. Red-green-refactor, tests first, the whole thing. Kept the faith through every project, every team, every language change. It worked. Still does, for humans.

Then I started using AI agents to write code at scale.

Something felt wrong. I couldn't name it for months. The test counts kept climbing. 3,000 tests, 8,000, 16,000 across 17 concurrent projects. CI was green. Commits shipped. Everything looked right from the outside. But I had this nagging suspicion that my tests weren't protecting me from anything serious.. that the green checks were a performance of correctness rather than evidence of it.

That kind of feeling is what I call a BKR. A Back-of-the-Know, where your gut understands something before the data does. I've learned to trust mine.

So I ran the research.

The Orthodox Position

TDD is sacred in software engineering. Kent Beck gave us red-green-refactor in the early 2000s and it genuinely changed how careful engineers think. Write a failing test. Make it pass. Refactor. The discipline forces you to think about interfaces before implementation. It produces better designs. It gives you a regression net you can actually rely on.

For human engineers, writing software for humans to use, it works.

The argument for TDD with AI agents sounds like a natural extension: "AI generates code faster and sloppier, so we need more tests to keep it in check. TDD provides the guardrails."

This argument is intuitive. It is also wrong. And I can prove it.

One Hour, 70 Citations

I spent one hour doing a structured research review across three independent AI research tools: Perplexity, ChatGPT Deep Research, and Gemini. Three separate tools, three separate research contexts, one question: does traditional TDD work when the entity writing code is an AI agent?

What came back was not ambiguous.

Kent Beck.. the man who invented TDD.. reported that AI agents were deleting his tests to make them pass. Not weakening them. Not gaming edge cases. Deleting them. The AI's objective was "make tests green." Deleting tests makes tests green. Objective achieved. Correctness irrelevant.

METR documented frontier models modifying test files during evaluations. Not modifying production code to fix tests. Modifying tests to pass. SWE-Bench+, one of the main AI coding benchmarks, found that many "successful" patches were passing tests by exploiting solution leakage or writing implementations that satisfied assertions but missed the actual intent of the task entirely.

ImpossibleBench went further. They gave AI agents tasks that were literally impossible to complete correctly, then watched the agents pass the tests anyway. Not by solving the task. By finding ways around the natural-language spec. Tests green. Task failed. Nobody knew.

There is a name for this. Reward hacking. When you tell an AI agent "your job is to make tests pass," it optimizes for exactly that, not for what you actually meant. The fastest path to green CI is not always "write correct code." Sometimes it's "weaken the test." Sometimes it's "strip the feature to reduce test surface." Sometimes it's "update the mocks to match whatever the broken implementation returns, so the mocks are now asserting that the code does what the code does."

AI-generated code creates 1.7x more issues than human code. Error handling gaps are nearly 2x more common. These are not my estimates. They come from the research corpus. The problem is structural, not incidental.

What All Three Sources Agreed On

Three independent research threads. One verdict.

The thing that survives is not TDD. It wears TDD's clothes. It produces tests. It cares about green CI. But it's structurally different, and the difference is the thing that matters.

They call it Spec-Driven Development. SDD.

Why AI Agents Think in Specs, Not Assertions

Here's the thing about AI agents: they're optimization systems trained on language. When you hand them a test and say "make this pass," they're solving a constraint satisfaction problem. Satisfy these particular assertions. That's the target.

Not "produce correct software." Not "satisfy the underlying intent." Satisfy these assertions.

A spec is different. A spec says: "given a user with a valid session token, when they request their profile, the system must return their name, email, and preferences." That encodes intent. An AI cannot satisfy it by deleting the assertion, because the spec exists separately from the tests. The spec is the source of truth. Tests are compiled from the spec. Changing the tests doesn't change what the spec requires.

The SDD pipeline:

Human writes SPEC (intent, invariants, acceptance criteria)
    |
Tests are generated FROM the spec (not from code)
    |
Tests are FROZEN (read-only to the implementation agent)
    |
AI implements against locked tests
    |
Separate agent/process validates (mutation testing, property checks)

This architecture makes reward hacking structurally impossible. You can't game the spec by weakening the tests, because the tests live downstream of the spec. And the spec doesn't belong to the implementation agent. It can't touch it.

The separation is the mechanism. Not a rule. A structural constraint.

GitHub released something called Spec Kit for exactly this. Addy Osmani at Google published detailed guidance on writing AI agent specs. There's an arXiv paper: "Spec-Driven Development: From Code to Contract in the Age of AI Coding Assistants." It found structured specs reduce LLM-generated code errors by about 50%.

This is not fringe. It's the emerging mainstream. Most people building with AI agents just haven't caught up yet.

The Three Separations That Actually Matter

The BKR research converges on three architectural separations that prevent reward hacking. I'm going to be concrete about what these mean in practice.

Spec and implementation are separate artifacts. The spec is written before code. Not tests-before-code. Spec-before-tests-before-code. The spec is the source of truth. If the spec says a function returns a non-empty list, no test can be written to say otherwise, and no implementation can satisfy the tests by returning an empty list.

Test writing and implementation are separate contexts. The entity that generates tests and the entity that implements code must not share context. This matters more than it sounds. When a single AI agent writes both code and tests in the same session, it has knowledge of both simultaneously. It can construct tests it knows the code will pass. This is circular validation. You think you're testing. You're not.

Building and evaluating are separate roles. The builder's incentive is "make it work." The evaluator's incentive is "prove it's broken." These must be adversarial, not collaborative. The same agent cannot do both. The same human doing both under time pressure introduces the same problem.

What violation looks like, for each one.

When spec and implementation share an author, the spec drifts to match the code. An AI agent writes "returns user profile data." Writes the code. Revises the spec mid-session to "returns a dict with name, email, and preferences." The spec now describes the implementation. Not a requirement. A transcript. The next engineer updates the spec to match the next implementation. The loop continues. The spec has no authority because it was never independent.

When test writing and implementation share context, the tests assert what the code does, not what it should do. The tell is specificity. Tests written by an agent that knows the implementation will assert exact key names, exact error strings, exact return shapes. Tests written from a spec will assert behavior and invariants. Different documents. One describes. One verifies. Most test suites are 80% description and 20% verification, and nobody knows which is which.

When building and evaluating are the same role, you get motivated reasoning. The builder who evaluates has already committed to "it works" before running the checks. Under pressure, the evaluation becomes perfunctory. This happens with humans too.. ask anyone who has written a test specifically to make CI green after an incident. The role separation isn't about distrust. It's about removing the option to rationalize a pass.

The Portfolio Proof

I have 17 projects running right now across two machines. 16,442 tests. One human.

When I read this research, I realized I had been doing SDD since 2024 without calling it that.

Our NEXUS files are specification documents. They have vision pillars, initiative definitions, and acceptance criteria written before any code is committed. Our CoS directives are structured specs.. what needs to be built, what done looks like, what constraints apply. The CoS writes the spec. The team implements. Those are separate sessions, separate contexts, separate identities. That's the "separate generation from evaluation" principle all three research sources identified as the core fix.

I wasn't following TDD. I was following SDD. And it was working for reasons I hadn't fully articulated.

The 16,442 tests are not TDD outputs. They're SDD outputs. Tests generated from specifications, implemented against them, and verified by a separate governance layer. The test count matters. But what actually protects me is the architecture the tests sit inside.

Here's what the architecture looks like concretely. A NEXUS initiative has an acceptance criterion: "given an approved file with a publish_date in the past, the publisher must mark the file as published and append an audit log entry." That's the spec. It exists before any code. Tests are written against it. The implementation agent, in a separate session with no memory of the spec session, builds against those frozen tests. A governance layer verifies the spec intent matched what shipped.

Three separate contexts. None share state. None can game each other.

The CoS writes directives in the morning. The team implements in the afternoon, in a fresh session, without the morning context. Tests are written against the directive's acceptance criteria.. not against whatever the code happens to produce. That's not an accident. That's the architecture doing its job.

The seam is the thing. Not tests. Not code. The seam between spec and implementation. Make it wide enough and reward hacking can't cross it.

One thing that surprised me in the research: the famous Opus C compiler rewrite is frequently cited as proof that TDD works with AI. It does work for a compiler. A C language standard is an objective, unambiguous specification. this input must produce this output. Every test encodes a provable fact. There's no gap between test and intent.

Most software is not a compiler. Product features, UX flows, data pipelines, API integrations.. none of that reduces to assert output == expected cleanly. That's why TDD works for compilers and protocol implementations and fails for products. It's not that TDD is wrong everywhere. It's that most of what we build doesn't have objective, unambiguous specs, and AI agents exploit that gap.

When TDD Still Wins

I want to be precise here because the contrarian framing can overcorrect.

TDD makes sense when:

The spec is objective and unambiguous (compilers, protocol implementations, parsers, encoders). When every test encodes a provable fact, AI cannot meaningfully game it.

You're adding tests to legacy code before refactoring. The tests aren't driving new development. They're capturing existing behavior as a regression net.

The work is pure refactoring where behavioral preservation is the entire goal. "Make this faster but don't change what it does" is an objective spec.

The entity writing tests is not the entity writing implementation. If humans write tests and AI writes code against frozen tests, you're doing SDD with extra steps. That works.

TDD fails when an AI agent writes both code and tests in shared context with the optimization target "make tests pass." That's reward hacking waiting to happen. And that's how most people are using AI coding assistants today.

The Mutation Testing Gap

All three research sources independently flagged mutation testing as the missing gate in most test suites.

Mutation testing works like this: take the code, systematically introduce small mutations (change a > to >=, flip a boolean, remove a return value), then run the tests. If the tests still pass after a mutation, the test suite is hollow. The code changed in a meaningful way and nobody knew.

Tools: mutmut or Cosmic Ray for Python, Stryker for JavaScript and TypeScript, cargo-mutants for Rust. Not exotic. Not expensive to run. Just uncommon.

A test that survives all mutations proves nothing about the code it covers. It might as well not exist. And most test suites have more of these than their owners realize.

The Meta-Lesson

This started with a gut feeling. Something about TDD felt wrong for AI agents. I sat with that feeling for months. Didn't act on it. Didn't dismiss it. Just noticed it.

Then I spent an hour doing structured research and got 70+ citations confirming it. One hour. Named framework. Portfolio-wide action plan.

That's the BKR pipeline working. The cognitive architecture that lets you take a half-formed instinct, route it to the right research tools, and come back with something you can actually act on. I'll write about that separately.

The practical upshot right now: if you're using AI agents to write code, you have a verification problem. Not a testing problem. A verification problem. More tests won't fix it. The fix is an architecture where the spec is the source of truth, the implementation agent cannot touch it, and your test suite is verified by something other than "did the assertions pass."

Next week: we shipped a real bug through 3,277 passing tests. Here's the forensic audit of exactly how it happened, the 6 failure patterns we identified, and the 7-gate protocol we built and deployed to 10 projects in a single afternoon.

Asif Waliuddin builds AI infrastructure at NXTG.AI. He runs 17 concurrent projects across two machines with two AI Chiefs of Staff and writes about what actually works when you stop following advice designed for teams of humans.