Insights•11 min read

The Test That Killed the Team: One awk Line, 12 AI Agents

October 14, 2026 by Asif Waliuddin

AI AgentsAI governanceSecurityThe Honor SystemAgent Control Plane

On October 14, 2026, an AI reviewer agent ran a cleanup test that contained one wrong awk line. Its kill list grew to almost every process on my machine, and kill -KILL took down 12 of my AI agents' sessions in the middle of their work. Nobody noticed for about two hours.

At 21:20:58 UTC on October 14, a review subagent on my agent server was re-checking a pull request for a plugin I build. Its job was narrow: confirm that the plugin cleans up its own processes. The harness it ran built the launcher's process tree with awk, then cleaned up with kill -KILL $tree. One line in the tree builder read a table entry where it should have tested membership. The list that reached kill held almost every process ID on the machine.

12 tmux sessions died, each one an AI agent whose Claude process was the pane's command. Two more agents lost their process but kept their session: my code-review agent, and the program manager agent that had commissioned the review. The test killed its own boss. Chromium went down with them, and so did the archiver that ships my memory database's write-ahead log. The kill stopped only when it reached the shell that was running the test.

What happened: an AI agent cleanup test with a computed kill list

The 12 sessions read like a roll call of everything I had running that evening. My product oracle agent. A product team agent about nine minutes into a tenancy rework it had started at roughly 21:12 UTC. Another product team agent in the middle of a pull request's test run. Three agents on my Forge products, one of them preparing the plugin's 3.14.0 release. My website team's agent. The content engine agent that drafts articles like this one. The podcast pipeline agent. An engagement agent. A program agent for my memory platform. And my own counterpart agent.

The quieter damage took longer to see. The write-ahead-log archiver restarts on its own but cannot attach, because the database still marks its replication slot as active for a stale sender process. WAL archiving for my memory database went down at 21:21 UTC, and when this was written it was still down, waiting for that sender to be cleared. The restore I describe below brought back each agent's conversation. It did not bring back work in flight, so the tenancy rework, the test run, the release prep and the program manager's open reviews all stopped where they were.

The awk line that turned a process tree into the whole machine

The harness tested t[pp[i]] and t[pi[i]] inside a condition, where t was meant to be the set of processes in the launcher's tree. In awk, reading an array element that does not exist creates it, with an empty value. The GNU Awk manual warns about this pattern and gives the fix: test membership with (indx in array), which creates nothing. In the harness, every process ID the condition looked up became a key in t, so the loop that printed the tree, for (k in t), printed almost every PID on the box.

I reproduced the mechanism with GNU Awk 5.1.0. You can paste both commands into any shell with awk installed. The first starts with one key, reads two entries inside a condition, and counts again. The second does the same check with the membership operator:

awk 'BEGIN{ t["root"]=1; pp["a"]="parent-of-a"; pp["b"]="parent-of-b";
  n=0; for(k in t) n++; print "before:", n;
  if (t[pp["a"]] || t[pp["b"]]) {};
  n=0; for(k in t) n++; print "after reading two parents:", n }'
# before: 1
# after reading two parents: 3
 
awk 'BEGIN{ t["root"]=1; if (("x" in t)) {}; n=0; for(k in t) n++; print "with (x in t):", n }'
# with (x in t): 1

Reading two parents inside an if grew the table from 1 key to 3. The membership test left it at 1. Scale that up to every process on a busy agent server and the "tree" becomes the machine. kill -KILL then worked through the list in awk's hash order, killing whatever came next, until it reached the shell that had launched it.

How I found it

I did not get an alert. Around 23:15 UTC I went looking for my product oracle agent's pane, and it was gone. So was a second product team's. Two missing sessions on a machine with a 15-minute self-heal should not happen quietly, so before anyone guessed at a cause I wanted the record of what had killed them.

Three records settled it. The reviewer's own transcript held the exact command, timestamped 21:20:58.989 UTC. The systemd journal at 21:21:00.566 UTC showed the WAL archiver and the desktop accessibility bus ending with code=killed, status=9/KILL, and the Chromium scope ending in the same millisecond. There was no kernel OOM entry, so memory pressure was ruled out: a process had sent SIGKILL. A diff of the 21:10 UTC tmux-resurrect save against a live tmux ls produced the list of 12 missing sessions. The incident record put the mechanism in one line: "kill -KILL on that list killed processes in hash order until it reached its own shell."

The self-heal was the part that hurt. The 15-minute self-heal and a guardian script exist to recreate sessions that disappear, and neither did so in two hours. Worse, at 21:30 UTC, nine minutes after the kill, the 10-minute tmux-resurrect save overwrote the last-known layout with the post-kill one. From that point on, my own backup described the damaged fleet as normal, and I found the problem by hand.

What I changed, and what is still owed

The restore finished at about 23:25 UTC, roughly ten minutes after I noticed. The controls are ordered, each with an owner.

1. Done: restored all 12 sessions with each agent's exact conversation. Each session came back in its own project directory with claude --resume <session id>, using the newest transcript in that project. A fresh session would have cost every agent its context; resume returned it. The program manager resumed its own conversation, and the code-review agent restarted.

2. Ordered: fix the harness to test membership. The condition becomes (k in t). A read inside a condition was the entire bug.

3. Ordered: kill test processes by their own process group. Start them under setsid, then kill -- -PGID, or run them in a systemd-run --user --scope unit. A negative PID sends the signal to every process in that group (kill(2)), so the test never has to rebuild its tree from a table. Never a computed PID list.

4. Ordered: add a refusal to the shell-command guard. The guard already runs in block mode on that machine. The new rule refuses kill or pkill with -9 or -KILL when the targets come from a variable or a command substitution, unless the target is a negative process group ID that the same command created.

5. Ordered: make the self-heal remember. It should recreate any session that existed in the previous save and raise an alert. The layout save must keep the pre-drop copy when sessions vanish, instead of overwriting the last good layout.

6. Ordered: clear the stale sender on the database. Confirm WAL streaming resumes, then report the exact gap window.

The restore had its own near miss. A coordination message from another agent landed in the program manager's pane while that pane was still a bare shell. Bash ran each line of the message as a command. Every line failed with a syntax error or "command not found", and nothing executed. As I write this, step 1 is done and steps 2 through 6 are owed. I am not counting any of them as a control until it has refused something in a test of its own.

The principle: a test is evidence, a refusal is a control

In Episode 5 of AI Unveiled, Policy Is Not a Control, I wrote that "A policy that cannot stop anything is a memo." The cleanup test existed to produce evidence of safety, proof that the plugin leaves nothing behind. The instrument producing that evidence was the hazard. Every agent in the chain would have agreed with a written rule like "tests only kill their own processes," and the awk line would still have run, because nothing between the agent and the kernel could say no.

Episode 5 splits governance into four loops, and this incident broke two of them. Control closure needs something that changes behavior before the action lands. The guard refusal does that: it reads the command before it executes and declines a SIGKILL aimed at a list computed at run time. Process groups make the honest target cheap to name, so the refusal costs a correct test nothing. Correction closure needs the last good state to survive the incident, and mine was overwritten nine minutes after the kill by the backup that was meant to protect it.

The same shape shows up anywhere an agent computes a target and then acts on it with no undo: a file list fed to rm -rf, a set of rows fed to a delete, a recipient list fed to a send. When the target is computed, the computation belongs to the blast radius. Name the target by construction. Refuse the computed one. Keep the last good state.

This is part of the Operator Series, field research from running AI-native workflows at scale. The governance model behind this fix is in Policy Is Not a Control: What Europe Just Taught America About AI Governance.

Asif Waliuddin is the founder of NXTG.AI. He writes about AI operations at nxtgai.substack.com.