A crash is never a pass. Three rules that kept 1,095 eval runs honest.

nickboy@users.noreply.github.com (Nick Liu) — Wed, 05 Aug 2026 00:00:00 +0000

The scariest failure mode in an eval harness is not a wrong answer. It is a harness problem wearing a model problem's clothes. A test file that does not run looks exactly like a problem no model can solve. A sandbox flake looks exactly like a failed attempt. Before I trusted any number from my CS1 auditing tool, I had to make those confusions impossible, and it came down to three rules.

Part 1 of this series reported scores from 1,095 sampled solutions. This post is about why I believe those scores. The harness lives in one Python module, and its docstring is a contract I wrote before the code:

Honesty rules:
 * The model is shown only `item.spec`; the hidden tests are written into the sandbox
 alongside the candidate but never given to the model.
 * A pre-flight check runs the *reference* solution against the hidden tests. If the
 reference does not pass, the item is malformed and the audit aborts (a HARNESS
 problem, never silently scored). This guarantees the test file is sound, so during
 the audit any candidate error (syntax, import, exception, failing assertion) is a
 legitimate FAIL for pass@k.
 * `HARNESS_ERROR` is reserved for runs where the sandbox produced no usable verdict
 (no report and no failing exit code). Such attempts are excluded from pass@k and
 reported separately, so they masquerade as neither pass nor fail.

Three rules. Each one exists because of a specific way a harness can lie.

Rule 1: run the reference solution first
#

Before any model attempt is scored, the harness runs the problem’s own reference solution against the hidden tests:

def preflight(item: Item, sandbox: Sandbox, limits: Limits) -> int:
 """Verify the reference solution passes the hidden tests. Returns the test count."""
 run = _run(item, item.reference_code, sandbox, limits)
 outcome, _passed, total, detail = _classify(run)
 if outcome is not Outcome.PASS:
 raise HarnessError(
 "pre-flight failed: the reference solution did not pass the hidden tests "
 f"({detail}). Fix the item before auditing.\n"
 f"stderr:\n{run.stderr[:2000]}"
 )
 return total

Without this gate, a broken test file is indistinguishable from a model that cannot solve the problem. You would conclude the problem is AI-resistant when your own tests do not run. In a project whose whole point was hunting for AI-resistant problems, this is the exact wrong conclusion to hand out for free. The gate turns that scenario into a loud abort instead of a quiet score of zero.

The gate also buys something subtle. Once the reference is known to pass, any error from a candidate solution (syntax error, import error, exception, failing assertion) is a legitimate FAIL. I do not have to wonder whose fault a stack trace is. The test file was proven sound minutes earlier.

Rule 2: three outcomes, not two
#

Most harnesses classify an attempt as pass or fail. Mine has a third verdict, and the classification code shows where it comes from:

raw = run.files.get(_JSON_REPORT)
if raw is None:
 # No machine-readable report. The test file is known-good (pre-flight), so a
 # non-zero exit means the candidate broke collection/execution -> FAIL. A clean
 # exit with no report is a genuine harness anomaly.
 if run.exit_code not in (0, None):
 return Outcome.FAIL, 0, 0, f"no json report; pytest exit {run.exit_code}"
 return Outcome.HARNESS_ERROR, 0, 0, "no json report produced"

With only two buckets, a crash has to land somewhere, and both landings are lies. Count it as a pass and your scores inflate. Count it as a fail and infrastructure flakes become model failures. HARNESS_ERROR attempts are excluded from the pass@k denominator and reported separately, so they masquerade as neither.

Across all 1,095 attempts, the harness error counter finished at zero. I want to be careful about what that proves. It does not prove the third bucket was unnecessary. It proves the infrastructure held, and I can only say that because the bucket existed to catch anything that did not. A counter that stays at zero is not vestigial. It is the evidence.

The scoring itself uses the unbiased pass@k estimator from the Codex paper rather than the naive “did any of the first k pass”, which is biased. That function validates k against the sample count and returns 1.0 when a pass is guaranteed by counting alone.

Rule 3: report what the sandbox does not guarantee
#

Every run record carries a capability string:

"capabilities": "network_isolated=True filesystem_confined=True memory_enforced=False"

The honest part is the False. macOS does not allow a hard memory cap the usual way, so the tool bounds memory by polling. It would have been easy to leave that field out, or worse, to print True because polling exists. Instead the record says: this boundary is best-effort. If a reviewer ever asks whether a candidate could have exceeded memory limits undetected, the answer is in the data, not in my memory of how the sandbox worked.

A full record, for shape:

{
 "item_id": "count-multiples-01",
 "model": "claude-haiku-4-5",
 "date": "2026-07-16",
 "n_scored": 20,
 "n_pass": 0,
 "n_harness_error": 0,
 "sandbox_backend": "local",
 "capabilities": "network_isolated=True filesystem_confined=True memory_enforced=False",
 "passk": { "1": 0.0, "5": 0.0 },
 "gate_k": 1,
 "theta": 0.2,
 "resistant": true,
 "construct_clean": true,
 "combined_verdict": "AI-RESISTANT (construct-valid)"
}

Model and date on every record. Part 1 showed the same problem scoring 0.00, 0.05, and 0.20 on three different days. Without the stamp, those would have been one contradictory number. With it, they are three measurements.

Lessons
#

A harness problem must never be silently scored. Abort loudly or record it as its own outcome, but never let it wear the model’s jersey.
Prove your test file with the reference solution before scoring anything. After that, every candidate error is legitimately the candidate’s.
Pass or fail is not enough. The third bucket exists so that crashes and flakes stop laundering themselves into your metrics.
Write down what your sandbox does not enforce. The embarrassing field in the record is the one that makes the record credible.

Part 3 covers the statistic that came back 0.00 when the agreement was 6 out of 8, and why the zero was correct.

References
#

Chen et al. (2021), “Evaluating Large Language Models Trained on Code” (the unbiased pass@k estimator)
Code shown above: src/cs1_auditor/harness.py and src/cs1_auditor/metrics.py in the cs1-auditor project, excerpted verbatim

Sandboxing on Nick Liu - Software Engineer

A crash is never a pass. Three rules that kept 1,095 eval runs honest.

Rule 1: run the reference solution first #

Rule 2: three outcomes, not two #

Rule 3: report what the sandbox does not guarantee #

Lessons #

References #

Rule 1: run the reference solution first
#

Rule 2: three outcomes, not two
#

Rule 3: report what the sandbox does not guarantee
#

Lessons
#

References
#