I made two sabotaged copies of a CS1 problem. Into the first I inserted zero-width characters, invisible in any editor. Into the second I went heavier and swapped letters for Unicode look-alikes as well. The hidden tests stayed byte-for-byte identical. If obfuscation works as an AI defense, the model's score should drop. It scored 0.85 on both copies. The attack did nothing. My fairness check flagged both copies anyway, and that second part is the one worth writing about. Auditing AI-Solvability · Part 4 of 5 1 2 3 4 5 Why sabotage my own problem # Instructors keep looking for ways to make assignments that AI tools fail. One family of ideas is to leave the problem alone and poison the text: invisible characters, homoglyphs, formatting tricks that a human reader never notices but that might derail a model reading the same bytes. Before trusting any resistance my tool (part 1) reported, I needed to know what this kind of tampering does to a score, because a problem that scores low for wording reasons is fake difficulty, not a real skill gap.
I labelled eight CS1 problems blind, before looking at any output from my auditing tool. Then I compared my labels with the tool's verdicts. We agreed on 6 of 8. Then I computed Cohen's kappa, the standard chance-corrected agreement statistic, and it printed 0.00. Exactly zero, sitting next to a 75 percent raw agreement. My first thought was that my code was broken. It was not. Auditing AI-Solvability · Part 3 of 5 1 2 3 4 5 The setup # This was the human-agreement check for the tool from part 1. The tool labels each problem AI-solvable or AI-resistant. For the check to mean anything, my own labels had to be committed before I saw the tool’s, so I wrote them down blind, sealed against hindsight. Eight problems, two raters: me and the tool.
The scariest failure mode in an eval harness is not a wrong answer. It is a harness problem wearing a model problem's clothes. A test file that does not run looks exactly like a problem no model can solve. A sandbox flake looks exactly like a failed attempt. Before I trusted any number from my CS1 auditing tool, I had to make those confusions impossible, and it came down to three rules. Auditing AI-Solvability · Part 2 of 5 1 2 3 4 5 Part 1 of this series reported scores from 1,095 sampled solutions. This post is about why I believe those scores. The harness lives in one Python module, and its docstring is a contract I wrote before the code:
For six weeks this summer I hunted for a CS1 programming problem that an AI model could not solve. I found exactly one candidate. It scored 0.00 against the weaker model, the stronger model solved it, and my fairness checks came back clean. Then I reworded the problem statement in plain English, changed nothing about the hidden tests, and the resistance evaporated. Auditing AI-Solvability · Part 1 of 5 1 2 3 4 5 What I was measuring # The tool is called cs1-auditor. It is a Python command-line tool that audits how solvable one CS1 problem is for a named model. You point it at a problem folder holding a plain-English spec, a hidden test suite, and a reference solution. It samples N candidate solutions from the model, runs each one in a sandbox against the hidden tests, and reports two things. The first is an AI-solvability score, which is pass@k for that model on that date. The second is a construct check, which asks whether any resistance comes from a real skill or from a trick in the wording. A problem only earns the label AI-resistant when the score is below a threshold and the construct check is clean.