Cohen's kappa came back 0.00. The agreement was 6 out of 8.

The setup

#

This was the human-agreement check for the tool from part 1. The tool labels each problem AI-solvable or AI-resistant. For the check to mean anything, my own labels had to be committed before I saw the tool’s, so I wrote them down blind, sealed against hindsight. Eight problems, two raters: me and the tool.

Raw agreement 6 of 8. For a first calibration pass on a small set, that felt fine. Kappa exists because raw agreement flatters you: two raters who both say “solvable” almost every time will agree often by luck alone, so kappa subtracts the agreement you would expect from chance. I expected it to shave 0.75 down to something modest. I did not expect it to shave 0.75 down to nothing.

Chasing the zero

#

The suspect list, in the order I worked through it: my kappa implementation, the label encoding, the library convention for two-category data. All innocent. The arithmetic is short enough to do on paper, and doing it on paper is what ended the confusion.

Over the eight problems, the tool produced one label. Every verdict was the same. The strong model solved everything it touched, so nothing crossed the resistance threshold, so the tool said “solvable” eight times. My blind labels said “solvable” six times and “resistant” twice. Observed agreement: 6 of 8, which is 0.75.

Now the chance correction. Expected agreement multiplies the raters’ label frequencies. The tool says “solvable” with probability 1.0. I say it with probability 0.75. So the chance of agreeing on “solvable” is 0.75, and the chance of agreeing on “resistant” is zero times something, which is zero. Expected agreement: 0.75.

Kappa is observed minus expected over one minus expected. That is (0.75 - 0.75) / (1 - 0.75). Zero, by construction, and not a rounding artifact. When one rater has no variance, every bit of observed agreement is exactly what chance predicts, and the correction consumes the entire signal. Kappa is not measuring how often we agreed. It is measuring how much better than chance we agreed, and against a constant rater, nobody can beat chance.

The step that made me believe it

#

I did not trust the explanation until I attacked it with synthetic labels. I generated fake rater pairs where one rater was constant and the other varied, swept the disagreement count, and watched kappa pin to zero every time regardless of raw agreement. Then I gave the constant rater a single deviant label and watched kappa jump to a nonzero value. The statistic was behaving exactly as defined. My data was the degenerate case, not my code.

If I had to keep one paragraph from this post, it would be that one. The temptation with a surprising statistic is to pick between two stories: “the number is broken” or “the number is profound”. Both are cheap. The third option costs an hour with synthetic data and tells you which world you are in. In my case the zero was neither broken nor profound. It was the statistic saying: this validation design cannot distinguish your tool from a coin that always lands heads, because on this problem set the tool never did anything else.

What the zero told me to do

#

A useless kappa on this set does not mean the agreement check was worthless. It means the set was wrong for the question. Every problem the strong model touched, it solved, which is itself the headline finding of the whole project. But it leaves the agreement study with no variance to correlate. The fix is structural: the set needed problems where the tool’s verdict varies, and by the end of the summer I had evidence those barely exist within CS1.

And the honest limit, stated in the body and not in a footnote: the second rater was me, and I built the tool. This was a self-check, useful for catching gross disagreement between my intuition and my own machinery. It is not external validation, and I do not present it as such. One rater, one vendor, one summer.

Lessons

#

• Kappa of 0.00 with high raw agreement is not a paradox. It means at least one rater never varied, and chance explains everything you observed.
• Before calling a surprising statistic broken or profound, spend the hour on synthetic data. Degenerate inputs produce exotic outputs from perfectly correct code.
• An agreement study needs variance in both raters. If your tool gives one answer on the whole set, you have learned something about the set, not about agreement.
• Self-checks catch self-delusion. They do not substitute for a second human rater, and writing that sentence in the report is cheaper than having a reviewer write it for you.

Part 4 puts the fairness checks under adversarial pressure: I hid zero-width characters inside a problem spec and measured whether any model flinched.

References

#

• Cohen (1960), “A Coefficient of Agreement for Nominal Scales” (the kappa definition used here)
• Messick (1995), “Validity of Psychological Assessment” (the validity framework this check belongs to)
• Run data: cs1-auditor .runs/*.json, pulled 2026-08-01