A Homebrew cask upgrade on 2026-08-28 left my `claude` binary hanging forever on `--version`. No error, no exit, just a prompt that never came back. For two days nothing looked wrong, because every session already open kept working perfectly. The first person to notice was the first person to open a new one. The signature was valid. The bytes were intact. A byte-identical copy of the file, sitting in the same directory, started instantly while the original hung. That left exactly one thing it could be, and it was not the file.
My dotfiles repo has a CLAUDE.md, and the CLAUDE.md has a rule in bold: every commit must pass the test suite, no exceptions. Within the first hour of an audit this July, I learned that this rule had been enforced exactly zero times since the day it was written. The hook file existed, its contents were correct, it even had its executable bit. It was just sitting at a path that yadm stopped reading a major version ago. No error message. No warning. To yadm, a hook in the wrong place and no hook at all are the same thing.
The scariest failure mode in an eval harness is not a wrong answer. It is a harness problem wearing a model problem's clothes. A test file that does not run looks exactly like a problem no model can solve. A sandbox flake looks exactly like a failed attempt. Before I trusted any number from my CS1 auditing tool, I had to make those confusions impossible, and it came down to three rules. Auditing AI-Solvability · Part 2 of 5 1 2 3 4 5 Part 1 of this series reported scores from 1,095 sampled solutions. This post is about why I believe those scores. The harness lives in one Python module, and its docstring is a contract I wrote before the code: