I went looking for AI-resistant CS1 problems. The search came up empty.

nickboy@users.noreply.github.com (Nick Liu) — Tue, 04 Aug 2026 00:00:00 +0000

For six weeks this summer I hunted for a CS1 programming problem that an AI model could not solve. I found exactly one candidate. It scored 0.00 against the weaker model, the stronger model solved it, and my fairness checks came back clean. Then I reworded the problem statement in plain English, changed nothing about the hidden tests, and the resistance evaporated.

What I was measuring
#

The tool is called cs1-auditor. It is a Python command-line tool that audits how solvable one CS1 problem is for a named model. You point it at a problem folder holding a plain-English spec, a hidden test suite, and a reference solution. It samples N candidate solutions from the model, runs each one in a sandbox against the hidden tests, and reports two things. The first is an AI-solvability score, which is pass@k for that model on that date. The second is a construct check, which asks whether any resistance comes from a real skill or from a trick in the wording. A problem only earns the label AI-resistant when the score is below a threshold and the construct check is clean.

Defaults: 20 samples per audit, pass@k reported at k=1 and k=5, threshold 0.2. The pass@k estimator is the unbiased one from the Codex paper, not the naive “did any of the first k pass”.

The scale of the evidence, so you can weigh it: I authored 58 problems, ran 34 of them against live models, and collected 1,095 sampled solutions between 2026-06-14 and 2026-07-23. Four model versions, all from one vendor: claude-sonnet-4-6, claude-haiku-4-5, claude-haiku-4-5-20251001, and claude-sonnet-4-5-20250929. One rater, one vendor, one summer. Every claim below sits inside that box.

The one candidate
#

Most problems never got close to resistant. On six clean baseline problems, the strong model went 1.00 across the board and the weak model slipped on only two (0.75 on a conditionals problem, 0.60 on a rounding problem). Five problems built around a non-default rule did better at separating the models, and one stood out:

Problem	The non-default rule	Sonnet	Haiku
count-multiples-01	closed range, negative bounds	1.00	0.00

Haiku produced twenty candidate solutions and zero passed. Sonnet solved it every time. The construct check came back clean: the skill was declared, the spec was plain ASCII, no tricks. The tool printed the label I had been chasing: AI-RESISTANT (construct-valid).

The reword protocol
#

A low score can mean the problem tests a real skill the model lacks. It can also mean the model tripped on the phrasing. To tell these apart I added a fifth check that is a protocol rather than a rule: reword the spec in plain English, keep the hidden tests byte-for-byte identical, and audit again. If the resistance is real, it should survive a paraphrase.

Probe	Haiku pass@1	Outcome
count-multiples-01 (baseline)	0.00	AI-RESISTANT (construct-valid)
reword 1	0.00	resistance held
reword 2	0.75	flipped to AI-SOLVABLE

The second paraphrase lifted Haiku from 0.00 to 0.75. Same tests. Same skill, supposedly. Sonnet solved the baseline and both rewords at 1.00 throughout. Whatever count-multiples-01 was measuring, it was not a stable property of the task. It was a property of one particular arrangement of words, on one model, on one date.

The score itself would not sit still
#

One more honesty requirement before the big claim. The 0.00 comes from the 20-attempt run of 16 July. I reran the same baseline problem on later dates:

Date	Attempts	Passed	pass@1
2026-07-16	20	0	0.00
2026-07-17	20	1	0.05
2026-07-18	5	1	0.20

So I will not write that Haiku failed it every single time. It scored 0.00 on the run I reported, and drifted to 0.05 and 0.20 on repeats. At first this annoyed me. Then I realized it is a second argument for the same thesis: even the score of one fixed problem moves between runs, so a single number pinned to a task is not a stable property. It is a measurement of a model, on a date, with a sample size attached. My tool stamps every record with all three, and this is why.

Widening the search
#

Maybe count-multiples-01 was unlucky and better candidates existed. I ran two more batches, 13 fresh problems designed around non-default rules. The bar for a keeper: fair by the construct check, solved by the strong model, failed by the weak model, and still failing after a reword. Not one problem cleared all four.

The most instructive failure was slice-1based-01. Haiku scored 0.15, which looked promising, but Sonnet also failed it at 0.00. A problem both models fail is not a resistant problem. It is a badly specified one. Its rewords lifted Haiku to 0.40 and 0.20, which fits the pattern: the difficulty lived in the wording.

Why the search may be structurally empty
#

The pattern generalizes into a tension I could not find a way around.

To be fair, and to be solvable by a strong model, the twist in a problem must be stated clearly. But a clearly stated twist is one that the weakest model you can still realistically use also reads. The genuinely weak models that a resistant problem would need have been retired. So the skill gap that resistance depends on has closed, at least for CS1, at least for the models I tested.

This does not mean no AI-resistant assessment can exist anywhere. It means that within my box (one vendor, four model versions, six weeks, 58 authored problems, 1,095 attempts) every candidate either failed the fairness bar or failed the reword. Prior work found GPT-4 passing about 95 percent of textbook CS1 tasks, so the ceiling was known to be high before I started. What I did not expect is how completely the floor had risen too.

Lessons
#

A low pass@k is a claim about wording until a paraphrase has failed to fix it. Reword first, then talk about skill gaps.
Report the run you measured, with its date and sample size. My “0.00” became 0.05 and 0.20 on later runs of the identical problem.
A problem that every model fails is not resistant. It is broken. Check the strong model before celebrating the weak model’s failure.
If a resistant problem needs an unclearly stated twist to work, the resistance is construct-irrelevant variance, not a skill.

The next post in this series is about the part of the tool that made these numbers trustworthy in the first place: the harness, and the three rules that kept 1,095 sandbox runs from lying to me.

References
#

Chen et al. (2021), “Evaluating Large Language Models Trained on Code” (the unbiased pass@k estimator, section 2.1)
Kiesler and Schiffner (2023), “Large Language Models in Introductory Programming Education” (GPT-4 solving 68 of 72 textbook introductory Python tasks)
Messick (1995), “Validity of Psychological Assessment” (construct-irrelevant variance, the concept behind the construct check)
Run data: cs1-auditor run records (.runs/*.json), pulled 2026-08-01; every figure above is from those records

Ai-Evals on Nick Liu - Software Engineer

I went looking for AI-resistant CS1 problems. The search came up empty.

What I was measuring #

The one candidate #

The reword protocol #

The score itself would not sit still #

Widening the search #

Why the search may be structurally empty #

Lessons #

References #