I hid zero-width characters in a CS1 spec. The model did not blink.

Table of Contents

Auditing AI-Solvability - This article is part of a series.

Part 1: I went looking for AI-resistant CS1 problems. The search came up empty.

Part 2: A crash is never a pass. Three rules that kept 1,095 eval runs honest.

Part 3: Cohen's kappa came back 0.00. The agreement was 6 out of 8.

Part 4: This Article

Part 5: Do not restyle a deck to look official. Wipe the template and keep its masters.

I made two sabotaged copies of a CS1 problem. Into the first I inserted zero-width characters, invisible in any editor. Into the second I went heavier and swapped letters for Unicode look-alikes as well. The hidden tests stayed byte-for-byte identical. If obfuscation works as an AI defense, the model's score should drop. It scored 0.85 on both copies. The attack did nothing. My fairness check flagged both copies anyway, and that second part is the one worth writing about.

Why sabotage my own problem
#

Instructors keep looking for ways to make assignments that AI tools fail. One family of ideas is to leave the problem alone and poison the text: invisible characters, homoglyphs, formatting tricks that a human reader never notices but that might derail a model reading the same bytes. Before trusting any resistance my tool (part 1) reported, I needed to know what this kind of tampering does to a score, because a problem that scores low for wording reasons is fake difficulty, not a real skill gap.

The donor problem was conditionals-01, a clean baseline that the weak model already passed at 0.75. I audited both sabotaged twins against claude-haiku-4-5, 20 attempts each, the same protocol as every other run in this series.

The attack failed
#

Problem	Model	pass@1	construct clean	verdict
conditionals-01-obf	claude-haiku-4-5	0.85	False	AI-SOLVABLE
conditionals-01-obf-heavy	claude-haiku-4-5	0.85	False	AI-SOLVABLE

Both copies passed at 0.85, slightly above the clean baseline’s 0.75 from an earlier run, which I read as ordinary run-to-run drift rather than obfuscation helping. Zero attempts were derailed by the invisible characters in any way I could detect. To be precise about the evidence base, because this is where sweeping claims get made: this is one problem, one model family, two obfuscation recipes, one summer. I am not writing “obfuscation cannot fool LLMs”. I am writing that these two attempts did not fool this model on these dates, and I did not find the result surprising after watching the same model shrug off paraphrases in both directions all month.

This is a negative result about the attack. I want to be straight about that. The experiment did not produce a defense, and I was not shopping for one.

The part that worked
#

The tool runs four offline fairness checks on every problem, no model call involved: the skill must be declared, the entry point must appear in the spec and be exercised by the tests, the spec must be plain ASCII with no hidden characters, and the layout must be free of construct-irrelevant obstacles. The third check exists precisely for this attack. Both sabotaged twins came back construct clean: False.

That flag matters even though the attack failed. Suppose a future model, or a weaker one, had stumbled over the invisible characters and scored 0.10. Without the check, the tool would have printed AI-RESISTANT, and I would have concluded I owned a problem that tests a real skill. The low score would have been fake difficulty, manufactured by tampering that no human student would even experience, since the rendered text looks identical. The construct check is the thing standing between “the model lacks this skill” and “I vandalized the input encoding”. A safeguard, not a headline.

The vocabulary for this comes from measurement theory: construct-irrelevant variance, score movement caused by something other than the skill you claim to measure. Invisible Unicode in a spec is about as construct-irrelevant as variance gets.

The detector connection
#

There is a sibling arms race on the detection side, and the literature on it shaped how I read my own result. Weber-Wulff and colleagues tested fourteen AI text detectors and all fourteen were easy to fool, with the worst errors landing on writers whose first language is not English. Sadasivan and colleagues argue AI text detection cannot be made reliable at scale. I am careful with both claims: fourteen specific detectors, tested in 2023, and I do not extrapolate to detectors released since. But the shape rhymes with my experiment. Tricks aimed at machines tend to wash out against the machines while the collateral lands on humans, and as someone writing in a second language, the collateral pattern in that first study is not abstract to me.

So my conclusion sits deliberately small. Obfuscating specs did not lower model scores in my two attempts, and any pipeline that reports AI-resistance should check its inputs for tampering before believing its own numbers, because the day the trick does work is the day it silently poisons your metric.

Lessons
#

Test the attack before trusting the defense. Two obfuscation recipes, twenty attempts each, zero effect on the score.
A fairness check earns its keep even when the attack fails. Its job is the counterfactual: the day a low score arrives, it tells you whether the difficulty is real.
Scope your negative results as tightly as your positive ones. One model family, one problem, two recipes. That is the whole claim.
Machine-versus-machine tricks have a track record of missing the machine and hitting people. Design assessments assuming the trick washes out.

Part 5 leaves the auditing tool and covers the other artifact of this project: a slide deck generator that steals its polish from the official university template, and the eight-inch hexagon gap it shipped with.