The scariest failure mode in an eval harness is not a wrong answer. It is a harness problem wearing a model problem's clothes. A test file that does not run looks exactly like a problem no model can solve. A sandbox flake looks exactly like a failed attempt. Before I trusted any number from my CS1 auditing tool, I had to make those confusions impossible, and it came down to three rules. Auditing AI-Solvability · Part 2 of 5 1 2 3 4 5 Part 1 of this series reported scores from 1,095 sampled solutions. This post is about why I believe those scores. The harness lives in one Python module, and its docstring is a contract I wrote before the code:
For six weeks this summer I hunted for a CS1 programming problem that an AI model could not solve. I found exactly one candidate. It scored 0.00 against the weaker model, the stronger model solved it, and my fairness checks came back clean. Then I reworded the problem statement in plain English, changed nothing about the hidden tests, and the resistance evaporated. Auditing AI-Solvability · Part 1 of 5 1 2 3 4 5 What I was measuring # The tool is called cs1-auditor. It is a Python command-line tool that audits how solvable one CS1 problem is for a named model. You point it at a problem folder holding a plain-English spec, a hidden test suite, and a reference solution. It samples N candidate solutions from the model, runs each one in a sandbox against the hidden tests, and reports two things. The first is an AI-solvability score, which is pass@k for that model on that date. The second is a construct check, which asks whether any resistance comes from a real skill or from a trick in the wording. A problem only earns the label AI-resistant when the score is below a threshold and the construct check is clean.
The first sign that a model generation had turned over under my feet was a billing prompt that made no sense. I ran /model to switch back to Fable 5 and the CLI opened a browser tab telling me to set up usage credits, for a model my Max plan already includes. A full restart of Claude Code fixed it. That little glitch sent me digging, and what I found was a pile of changes, some breaking, some silent, scattered across at least five official pages. This is the consolidated list I wanted that week. 🧪 Tested with Claude Code 2.1.220 · macOS Eight weeks that rearranged the lineup # 06-09
Fable 5 ships
My Claude Code config now holds two hooks, ten skills, and three custom subagents, and most of them started life in the wrong layer. The instruction the model followed nine times out of ten lived in a prompt until I accepted that nine out of ten is a coin I lose every day. The workflow I pasted into chats became a skill. The bulk work that was draining my priciest model's quota became a fleet of cheaper agents. Same features, different failure modes. Taming Claude Code Sessions · Part 6 of 6 1 2 3 4 5 6 🧪 Tested with Claude Code 2.1.x · macOS The four layers # Claude Code has four extension points, and they answer four different questions:
I run four or more Claude Code agents at once, and until recently they all shared one working tree. Two agents editing the same repo means one of them eventually builds against the other's half-finished changes. Git worktrees fix that cleanly. What nobody warned me about is that the fix multiplies a different problem I already had: forgetting which folder a session lives in. Taming Claude Code Sessions · Part 5 of 6 1 2 3 4 5 6 🧪 Tested with Claude Code 2.1.x · macOS The symptom # With several agents in one directory, the working tree is shared mutable state. Agent A refactors a partial, agent B runs the build, and B’s “failure” is really A’s work in flight. I had been dodging this by scoping agents to different subdirectories, which works until it does not.
For six months I assumed nobody could tell whether anyone read this blog, because I had never added analytics. Wiring up Cloudflare Web Analytics by hand taught me two things: the obvious place to paste the snippet would have shipped my Playwright suite's page views into the dashboard, and the dashboard had been quietly counting my visitors for two months anyway. 🧪 Tested with Hugo 0.163.3 · Blowfish 2.104 Publishing into the void # The site’s hugo.toml had a googleAnalytics line commented out since roughly the first commit. I never uncommented it. GA4 wants a cookie disclosure, ships a chunky client, and ad blockers eat it anyway, which felt like a lot of ceremony for a personal blog whose one open question was “does anybody visit.”
Paste one of my post links into a social preview and the card comes up with no image. The site itself renders fine, every page, every browser. The culprit was one character in `hugo.toml`: `baseURL = "/"`, which quietly turns every absolute URL the site emits into a relative one that only a browser can love. 🧪 Tested with Hugo 0.163.3 · Blowfish 2.104 The symptom # Share cards without images, that was the visible part. View source on any page and the metadata told the fuller story:
I changed one sentence in a blog post, opened a pull request, and watched CI spend about 85 seconds installing a headless Chromium to confirm my prose still turned into HTML. The obvious fix, telling the workflow to ignore content paths, would have quietly broken every merge instead. Symptom: a browser for a one-line edit # Every pull request on this blog runs two jobs: a lint job, and a build-and-test job. The second one builds the site with Hugo, link-checks the output with htmltest, then installs a headless Chromium and runs a Playwright suite against a live hugo server. End to end, roughly 85 seconds, and almost all of it is Playwright.
If you use tmux, you've hit this: ten windows open and they're all named `zsh` or `node`. Which one had your AI agent running? No idea. Let's make tmux label windows usefully. Taming Claude Code Sessions · Part 3 of 6 1 2 3 4 5 6 🧪 Tested with Claude Code 2.1.x · macOS / Linux New to tmux? It’s a “terminal multiplexer”: it splits one terminal into many windows and panes that survive disconnects. The only vocabulary you need here: a window is like a browser tab inside tmux; the bar at the bottom lists them. The prefix is the key you press before a tmux command, commonly Ctrl+b (mine is Ctrl+a). Why everything is named zsh # By default tmux has a setting called automatic-rename turned on. It renames each window after whatever program is running in it. A shell? zsh. A Node program (like Claude Code)? node. Helpful in theory, useless when everything collapses to the same word.
Once you're comfortable with AI coding agents, you start running several at once: one refactoring here, one writing tests there, one stuck waiting for your approval. Keeping them straight is its own little skill. Taming Claude Code Sessions · Part 4 of 6 1 2 3 4 5 6 🧪 Tested with Claude Code 2.1.x · macOS / Linux Here are two ways to do it: a lightweight tmux plugin, and (briefly) dedicated “AI terminal” apps.
You're deep in a great Claude Code conversation. You close the terminal. The next day you want to pick up where you left off… and you can't find it. Sound familiar? Let's fix that. Taming Claude Code Sessions · Part 1 of 6 1 2 3 4 5 6 🧪 Tested with Claude Code 2.1.x · macOS / Linux What is a “session,” and where does it go? # Every time you run claude, you start a session, one conversation, with its full history. When you quit, that history doesn’t vanish. Claude Code saves it to disk, organized per project folder, here:
In Part 1 we learned that named sessions are easy to find. Now let's make naming automatic with a hook. This is also a perfect first hook project, so I'll explain the whole idea from scratch. Taming Claude Code Sessions · Part 2 of 6 1 2 3 4 5 6 🧪 Tested with Claude Code 2.1.x · macOS / Linux Part 1 left us with one chore: you still have to remember to name your sessions. Let’s delete that chore.