Cohen's kappa came back 0.00. The agreement was 6 out of 8.
··1026 words·5 mins
I labelled eight CS1 problems blind, before looking at any output from my auditing tool. Then I compared my labels with the tool's verdicts. We agreed on 6 of 8. Then I computed Cohen's kappa, the standard chance-corrected agreement statistic, and it printed 0.00. Exactly zero, sitting next to a 75 percent raw agreement. My first thought was that my code was broken. It was not. Auditing AI-Solvability ยท Part 3 of 3 1 2 3 The setup # This was the human-agreement check for the tool from part 1. The tool labels each problem AI-solvable or AI-resistant. For the check to mean anything, my own labels had to be committed before I saw the tool’s, so I wrote them down blind, sealed against hindsight. Eight problems, two raters: me and the tool.