Part of my research on robust evaluation of adaptive systems. Branch closed 2026-09-29; the reading of its one clean held-out pass was narrowed 2026-10-02 after an external reproduction — see R3.
AI Evaluation · Agent Reliability · Experimental Systems
Evaluator Assurance: What Survived
Published Sep 29, 2026 · Updated Oct 2, 2026I tested seven plausible ways to make evaluator-driven AI systems more trustworthy across AgentRewardBench and τ-bench. Most weakened or failed when exposed to preregistration, held-out data, operational metrics, cheap baselines, and — for the one rule that passed its numeric gate cleanly, R2's acceptance being underpowered — an outside reader reproducing the result. None of the seven came through as a robust improvement. What is left is a smaller, better-evidenced claim about what evaluator assurance does not buy.
This is a falsification study, not a framework proposal. The research value comes from determining what did not earn authority, and preserving enough evidence that a skeptical reader can inspect why.
3,086
trajectory records across two studies
5,530
analysed repeat judgments (R=5)
$49.50
billed inference spend
7
mechanisms frozen and tested
13 + 11
process errors: found in-study, then by external reproduction
4-day
intensive research sprint
The question
LLM-backed systems increasingly use evaluators to decide whether an agent succeeded, whether to retry, whether to escalate, or whether to trust a generated result. When those evaluators make mistakes, the intuitive response is often: sample more, add another judge, add routing, add deterministic guards, combine everything.
The project asks a more precise question:
Which of those mechanisms actually earn their cost and authority when tested against real trajectories, with preregistered gates and a corpus whose outcome labels were withheld until the rule was frozen?
Corpora
| Corpus | n | Evidence class |
|---|---|---|
| AgentRewardBench rev b6d17e6 | 1,106 | Preregistered |
| τ-bench historical_trajectories/ | 1,980 | Fresh / label-blind |
Corrected 2026-10-02. τ-bench was described as “never touched” here. It was not: its trajectories were read during RC1's scoping, with the reward and info fields withheld in code so the outcome labels stayed unseen (docs/required_conjunct_scoping.md §8.4). That is a label-blind hold-out, which is weaker than an untouched corpus and stronger than a re-used one — and it is what the gates were actually run against.
Mechanisms tested
- M1Repeated execution of the same judge
- M2Alternate-evaluator pairing / escalation
- M3–M6Deterministic pre-check rules (R1–R4, held-out)
- M7Static required-conjunct matching (RC1, fresh corpus)
Result 1 — Preregistered
Repetition had almost nothing to average over
gpt-4o-2024-11-20 · temperature 0.0 · seed 0 · judge stage only
Five repeated calls were made on each of 1,106 AgentRewardBench trajectories — 5,530 calls total. If repeated sampling were recovering meaningful variance, a substantial fraction of cases would land in the middle of the k=0–5 distribution.
The deployed judge was not perfectly deterministic. It was simply stable enough that repeated sampling had almost no useful error to recover. Only 20 of 1,106 cases showed any intermediate correctness across five draws.
20
of 1,106 cases had k = 1–4
Marginally worse
majority-of-5 vs single call, both error directions
−39.7% calls
first-to-3 stopping; identical verdicts to majority-of-5
What this licenses: for this deployed configuration, five-repetition majority voting was not worth the cost. First-to-3 stopping recovered all available savings. What this does not license: unscoped claims about repetition. Provider-side nondeterminism is not excluded by five draws, and temperature-0 is not the same as no variability. Scope is one judge, one corpus, judge stage only.
Corrected 2026-10-02
The call saving read 39.8% here until an external reproduction recomputed it from the raw calls. It is 39.7%, not 39.8% — and the fraction behind it was also stated backwards here: majority-of-5 costs 5,530 calls and first-to-3 uses 3,336 of them, so the saving is the 2,194 calls not made — 2,194 of 5,530. The 0.1-point error changes nothing, and that is the uncomfortable part — it was a hand-typed literal sitting next to a script that computed the value correctly, and nobody compared them. The study's spend was also reported as $46.18 throughout; that is the cost of the 5,530 analysed calls, while $49.50 was actually billed, because 356 further valid calls were paid for and never entered the analysis.
Result 2 — Exploratory
The attractive metric was not the executable quantity
The natural way to evaluate an alternate judge is to ask: when the primary judge is wrong, how often does the alternate get it right? That quantity — P(alt correct | primary wrong) — looks like operational recovery. It is not.
A runtime policy does not know whether the primary is wrong. It sees two judgments and must decide whether to overturn. The reference-conditioned quantity conditions on a label unavailable at runtime, which is why it runs about ten points higher than the precision a deployed policy would actually achieve.
Missed failures · the primary passed a failing run
Reference-conditioned catch rate 0.469 (15/32)
Operational overturn-to-fail 0.366 (15/41)
10.3 points
False alarms · the primary failed a passing run
Reference-conditioned rescue rate 0.562 (73/130)
Operational overturn-to-pass 0.465 (73/157)
9.7 points
The gap is about ten points in each direction. A report that shows only the reference-conditioned figure makes alternate-judge escalation look more useful than a deployed policy would find it. Primary errors split 130 false alarms to 32 missed failures, so the pooled recovery rate of 0.543 is dominated by the easier-to-recover stratum — which is also why it cannot be differenced against either operational precision.
Corrected 2026-10-02
This section read “the gap is 20–30 points” until an external reproduction asked what the subtraction was between. 0.543 is recovery pooled over both error directions; 0.366 and 0.465 are direction-specific. Subtracting across that axis compared two different populations. Matched by direction the gap is 10.3 points and 9.7 points, not 20–30 points. The finding survives, and the split is what makes it survive: across all eight alternates the false-alarm-side gap is positive throughout (+9.7 to +46.0 points) while the missed-failure side spans −14.7 to +23.1 points and is negative for two of them. So “the deployable quantity is lower” is true of false alarms generally and is not true of missed failures generally. The pooled phrasing concealed a sign change — the same error class as R3's confound, in the section about conditioning on the wrong thing.
Predeclaration changed the story
The predeclared evaluator pair (selected before joint outcomes) reached 0.543 pooled recovery. The best post-hoc result was 0.759 (123/162) — in an identically structured report. That best was a tie between two alternates, gpt-4o-mini-noscreen (axtree) and qwen-2.5-vl-noscreen, on exactly the same count; this page called it “the best post-hoc pair” until 2026-10-02, which is the phrasing that makes post-hoc selection sound like a discovery rather than a coin landing twice. The predeclared pair ranked last of eight candidates on pooled recovery.
If evaluator selection and metric choice happen after looking at the matrix, it is easy to tell yourself a convincing story. The pair that was locked before outcomes ranked tied first of eight on missed-failure recovery (0.469, 15/32 — three candidates share the value exactly; this said four until 2026-10-02) — the operationally relevant stratum — which was only visible because the choice was already frozen.
Evidence class: pair selection is preregistered (class A); the operational decomposition was conducted after outcomes were visible (exploratory). Both facts are stated.
Result 3 — Held-out validation
Deterministic rules needed their own held-out test
Four deterministic pre-check rules (R1–R4) were preregistered and frozen before evaluation on a 1,260-case held-out arm (1,259 scored; one case had no parseable evaluator verdict). Each fires on production-visible inputs only — no oracle access. Their results were heterogeneous.
Only two of the four carried a numeric accept/reject threshold. §7 of their preregistration gives R1 and R4 a direction and no bar at all, on the stated grounds that neither changes a verdict. That was misread on this page until 2026-10-02: R4 was shown as a failure against a threshold it never had.
| Rule | What it checks | Disposition | What it taught |
|---|---|---|---|
| R1 | Self-contradictory infeasibility claim: the agent calls report_infeasible while asserting completion | DIRECTION FAILEDno numeric gate (§7) | 0.51× — the signal points the opposite way to its specification; 17 of 27 firings are true successes. Had it been given veto power it would have helped 6 and harmed 14. The escalation half survives: 33.3% evaluator error on firings (9/27) vs 15.2% elsewhere (187/1,232), and unlike R3 it fires across three benchmarks and stays positive inside each one. Underpowered — the fired arms are 7, 10 and 10 cases against unfired arms of 283, 363 and 458 — and so not obviously confounded. (This read “every within-slice arm is under 20 cases, not confounded” until 2026-10-02: only the fired arms are small, and three same-signed slices at n=7–10 are not evidence that no slice effect is there.) |
| R2 | Negative self-report on an imperative modification goal | ACCEPTED / UNDERPOWERED | 13 eligible cases, 6 verdicts changed — 6 helped, 0 harmed. Meets its preregistered criterion. Firing precision is 1.06× and the rule-of-three harm bound at n=6 is 39.3%. The acceptance stands; the magnitude is not trustworthy. What it licenses is a larger test, not deployment. |
| R3 | Evidence-gap detection: goal references an image the evaluator cannot observe | PASSreading narrowed 2026-10-02 | Passed its preregistered test at 25.6% evaluator error on 133 fired cases vs 14.4% elsewhere, then was narrowed on reproduction: all 133 firings are one benchmark, and it is the benchmark with the highest judge-error point estimate (22.4%, next-highest 19.8%). Within that slice, 25.6% vs 19.7% (p = 0.26). Details below. |
| R4 | Terminal search-results route | DIRECTION METwas shown as FAIL until 2026-10-02 | 1.20× enrichment for reference-fail — which is the only criterion §7 gave it, and it met it. Evaluator error is 16.1% on firings (19/118) vs 15.5% elsewhere (177/1,141), a 0.6-point difference (p = 0.89), so the rule carries no information about evaluator reliability; but that contrast is not a preregistered test, and failing it is a post-hoc judgment, and this page asserted it as the disposition. The accurate statement is narrower and less useful: R4 met its direction and taught nothing. |
R3 in detail: it passed its test, then the test turned out to be confounded with its benchmark
R3 fires when the evaluator receives a goal that references an image it cannot observe. The practical interpretation: return UNVERIFIABLE and escalate to human review rather than pretending PASS or FAIL is well-supported. Evaluator error on the 133 fired cases was 25.6% vs 14.4% elsewhere, Fisher p = 0.0015. That was the one rule that passed its numeric gate cleanly — R2 was also accepted, but underpowered — and this page said so until 2026-10-02.
Narrowed 2026-10-02 — found by an external reproduction audit, not by me
An outside reader asked a question the preregistered analysis never asked: which benchmarks does R3 fire in? All 133 firings are visualwebarena. Zero in the other three. And visualwebarena also carries the highest judge-error point estimate of the four — 22.4%, against 19.8% on webarena, 11.1% on workarena and 3.9% on assistantbench. So the pooled test was not comparing escalated cases against unescalated ones. It was comparing one benchmark against three others, one of which is barely easier. (This listed only assistantbench and workarena until 2026-10-02, and called visualwebarena the slice “where the judge is weakest”. Omitting webarena turned a 2.6-point gap over the next-highest slice into an apparent 22.4-against-11.1 contrast, which overstates how unusual the slice R3 lives in actually is.)
| Comparison | Fired | Not fired | Separation | Fisher p |
|---|---|---|---|---|
| Pooled, as preregistered | 25.6% (34/133) | 14.4% (162/1126) | +11.2 pp | 0.0015 |
| Within visualwebarena | 25.6% (34/133) | 19.7% (31/157) | +5.8 pp | 0.26 |
| …missed failures only | 25.3% (22/87) | 15.8% (18/114) | +9.5 pp | 0.11 |
| …false alarms only | 26.1% (12/46) | 30.2% (13/43) | −4.1 pp | 0.81 |
The last two rows are not preregistered and were chosen after the pooled reading failed, so read them as exploratory. They say what the surviving part is about: “evaluator error” pools two mistakes a reviewer acts on differently, and all of the residual sits on the missed-failure side. Where the reference says the agent failed, R3's cases carry more missed failures than the rest of the same benchmark. Where the reference says it succeeded, the sign reverses. Neither stratum is significant on its own. That is the shape an absent image premise should produce — the judge cannot see the evidence that would reveal the failure — which makes it a more specific hypothesis to test next, not a rescue of this one.
Of the +11.2 pp that was published, +5.8 pp survives conditioning and +5.4 pp is slice identity. The residual runs in the direction R3 predicted and is underpowered on both sides of the within-slice comparison, so the rule is unproven, not refuted — settling it needs a corpus where the evidence gap occurs outside a single benchmark. The preregistered disposition does not move, and deliberately so: R3 met its criterion and the test ran as specified, so relabelling it now on an analysis the preregistration never named would be the same post-hoc move this study criticises everywhere else. What is narrowed is what the pass was taken to mean. Reproduce both rows with scripts/arb_r3_slice_check.py.
How wide is the damage? R1's escalation half and R4's comparison are the same kind of pooled error-rate claim, so all four rules got the same check rather than just the one that was challenged. R3 is the only rule of the four whose firings sit in a single benchmark, so it is the only one whose pooled contrast can be a slice artefact. R1 fires across three benchmarks and keeps a positive residual in each; R4's residuals change sign between slices, which is what a rule carrying no information about evaluator reliability should look like — consistent with R4 carrying no evaluator-level signal. (This read “and matches the FAIL it already had” until 2026-10-02, which contradicts R4's actual preregistered label: §7 gave it a direction and no numeric bar, and it met that direction.) The correction is one rule, not the rule set — established by running the check, not by asserting the scope.
The hazard was already written down, twice, in two different files — though not where this page used to say. It is not in the preregistration. It is a caveat in the repair results write-up, added after R3's numbers were in: “Any rule that escalates the cases a judge finds hard will pass a test of the form ‘is the judge worse on the escalated subset’. That test is necessary, not sufficient, and the preregistration should have said so.” The last clause is the whole of it: the preregistration did not say so, and Disposition: ACCEPTED appears two paragraphs below that caveat. The synthesis separately records “visualwebarena-only firings” in a scope column. Neither document knew about the other. (Until 2026-10-02 this page credited the sentence to the preregistration, which made the project look as though it had foreseen the hazard and merely failed to act on it. It had not.) The defence available at the time — that R3's condition is structural, uses no oracle access, and was frozen before the judge's per-slice error rate was known — is all true, and none of it helps: freezing a rule in advance does not control for a covariate the test never measured.
The quarantine could not have caught it either, and it is worth being precise about why. The split held back the cases that suggested the rules, so no rule was tested on its own discovery set. But 1,064 of the 1,260 held-out cases (84.4%) were already inside the population whose judge correctness this project had tabulated benchmark-by-benchmark in an earlier study. Per-benchmark difficulty was knowable when the rules were written. A held-out split controls for having seen these cases' outcomes; it does not control for having seen the shape of the corpus, and a confound that lives in a covariate survives any split that does not stratify on it.
The caveat that was already on this page still holds, and it was always the larger one: 99 of the 133 escalated cases were already correct. R3 was never a correction mechanism, only an error-concentration signal whose value is a function of human review cost. What the audit removed was the evidence that it concentrated error beyond what its benchmark explains — not the evidence that it concentrated error at all, which is what this sentence claimed until 2026-10-02 and is a stronger statement than the within-slice residual supports.
What this does and does not change about the pattern
The structural argument is unharmed. A rule that asserts a gap in the evidence — rather than one that claims to know the verdict — is still easier to validate and harder to get subtly wrong. R1 and R4 tried to predict outcomes and failed at it; R1's weaker half, the part that only flags a case for review, is the half that survived. R3 makes only that weaker kind of claim.
What changed is that the pattern now has no clean confirming instance in this study. It is a hypothesis about what makes assurance rules validatable, held on reasoning rather than on a result — which is a weaker thing than this page previously claimed, and worth saying plainly instead of leaving R3 in place to carry the argument.
Result 4 — Preregistered / Fresh corpus
A good extractor does not make a bad rule good
RC1 was a static required-conjunct detector: given a task, extract the observable actions an agent must perform, fire if any are missing, and let the evaluator'sverdict stand otherwise. Three gates were preregistered before any τ-bench outcome label was read — the corpus itself was examined label-blind during scoping:
- A1: firing volume must be <15% (prevents the detector from dominating evaluation)
- A2-traj: trajectory lift must be ≥1.50×
- A3: label-blind extractor precision must be ≥27/30 — precision only; the gate never asked what the extractor missed
A1 — FAIL
25.3%
vs <15% ceiling — 501/1,980 fired
A2-traj — FAIL
1.165×
vs ≥1.50× bar — 1.101× within-task
A3 — PASS
25/27
extractor precision, out of sample (28/30 as scored, bar ≥27/30)
Corrected 2026-10-02
A3 was reported here as 28/30 with no qualification. An external reproduction checked which cases those thirty were: three of them (02, 06, 16) are exactly the cases commit A2 had already repaired before pass 2 scored them, so 28/30 is not out-of-sample. Excluding them leaves 25/27, and this note called that “a weaker pass, still a pass” until later the same day. It is more awkward than that: 25/27 is 92.6%, which clears the ≥27/30 bar as a rate (90.0%) but falls below it as a count — and the preregistration states the bar as a count. A3 is also precision only: it asks whether the conjuncts the extractor emitted were correct and never whether it missed any. So “the extractor worked” is established against false positives and says nothing about recall.
The extractor's output was precise. The rule it fed did not work. RC1 failed both binding gates on the fresh corpus — not because the detector was parsing incorrectly on the conjuncts it found, but because firing on 25% of trajectories and lifting only 1.165× means the conjunction condition was too permissive to isolate the cases where the evaluator actually needed help. Whether a more complete extractor would have fired differently is not something A3 can answer.
What the oracle established — and did not
![Bar chart comparing within-task lift pooled across tasks, for three conditions with bootstrap confidence intervals: oracle reconstruction 2.339× [1.927, 2.927] on 94 tasks, null rule 1.171× [0.802, 1.652] on 80 tasks, and RC1 production detector 1.101× [0.903, 1.360] on 109 tasks. The null and RC1 intervals overlap heavily; each bar is computed on a different task set. The stratified Mantel–Haenszel estimates are smaller and are given in the text.](/projects/evaluator-assurance/fig3_lift_comparison.png)
A reconstruction using benchmark-authoritative actions showed that the latent construct carried signal RC1 failed to recover: the Mantel–Haenszel risk ratio stratified by task puts the oracle at 2.054 [1.752, 2.530], against 1.237 [0.904, 1.650] for the null rule and 1.150 [0.979, 1.352] for RC1 — neither of which excludes 1. Pooled rather than stratified the oracle reads 2.339× [1.927, 2.927], which is the figure plotted in Figure 3 above; the ordering also survives the paired test, oracle minus null being +1.177 [+0.721, +1.687]. None of this is the same as showing a deployable detector exists — the oracle uses inputs a runtime evaluator cannot access. Building one would require new representation work that was not attempted.
A null rule requiring no obligation model at all ("did the agent write anything?") reached 1.171× [0.802, 1.652] against RC1's 1.101× [0.903, 1.360]. Those two are not separated: paired on the same task resamples the difference is +0.078 [−0.327, +0.575], which spans zero. The null rule was preregistered as disqualified before any computation (volume 24.1% fails A1), so it does not compete with RC1 as a mechanism. It functions as a floor, and RC1 neither cleared it nor fell below it.
Corrected 2026-10-02
This section said the null rule came in “above RC1's 1.101×”, and the project's own synthesis said RC1 fell below a no-model baseline. Neither is supported. The two point estimates are computed on different task sets — 80 and 109, because the within-task filter keeps only tasks where that rule both fires and does not — and paired properly the difference is +0.078 [−0.327, +0.575]. The ordering was read off two bare numbers. The conclusion that RC1 does not earn deployment stands on the two binding gates it actually failed, A1 and A2-traj, not on this comparison.
Why the branch was deliberately closed
RC1 failed both binding gates — A1 on firing volume and A2-traj on lift. The oracle establishes the construct is real. But "the construct has signal" and "a deployable detector can be built" are different claims, separated by representation-design work this branch did not complete. Extending the branch without new representation work would be changing the research question, not answering it. Note what this does not rest on: RC1 is not shown to be worse than a null rule, and A3 only establishes that the conjuncts it did emit were right.
The research changed its mind
Two counts, kept apart. 13 process errors were found during the study by its own safeguards and audits — 6 before the affected experiment's outcomes were visible, 7 afterward. 11 more were found by a separate end-to-end reproduction run from outside the project on a frozen copy, none of them caught by any safeguard here.That is 24 rows in the audit table, but the sum is not the interesting figure: the safeguards that ran ahead of the data caught 6 of the 13 this project found about itself, and 11 it did not find at all.
Corrected 2026-10-02
This section said “Thirteen process errors were documented across 62 commits. Six were caught before the affected experiment's outcomes were visible; seven were caught afterward.” That count was right and is unchanged — it is the internal one. What it did not have was a second count beside it. A separate reproduction then found 11 more: ten wrong or over-strong summary figures and protocol descriptions, and the benchmark confound below. None of the 11 needed new data — every one was recoverable from the published numbers. By class, against the audit table's own rows 14–24: two restated a value a script had already computed correctly (rows 14, 15), three compared rates across incompatible conditionals (17, 20, 24), two dropped a condition the method required — a cost weighting, and the analysed-versus-billed distinction (16, 18), two asserted an ordering or an extremum that was never checked (19, 23), one described a blind as stronger than it was (21), and one cited a precision audit that had scored its own fixes (22).
2026-10-04
Four further wording corrections in the repository (see docs/research_synthesis.md §10, pre-submission claim check). On this site: the corpus card for this study summarised it as a result that survived until 2026-10-02 and carried the superseding paragraph itself until today — that paragraph is now here, where the rest of the record is. The reproduction that found eleven of the errors above is described as separate rather than independent throughout, because it was author-directed and AI-assisted. And the oracle's within-task lift is now labelled as the pooled estimator wherever it appears, with the task-stratified Mantel–Haenszel ratio named first.
Benchmark confound in the one clean numeric pass (found externally)
R3's preregistered test pooled evaluator error across four benchmarks. All 133 of its firings are in one of them — the one with the highest judge-error point estimate (22.4%, against 19.8% on the next-highest slice) — so the test compared one benchmark against three others rather than escalated cases against unescalated ones. Conditioned on slice, the separation falls from +11.2 pp (p = 0.0015) to +5.8 pp (p = 0.26). Both facts needed to see this were already in the repo, in two different documents that never referred to each other. R3's preregistered disposition is unchanged — the test ran as specified and met its criterion, and relabelling it afterward on an analysis the preregistration never named would be the same move in the other direction. Only the reading is narrowed.
The metric gap was differenced across error directions (found externally)
The headline gap between reference-conditioned recovery and operational overturn precision was published as 20–30 points. It came from subtracting two direction-specific precisions from a recovery rate pooled over both directions. Matched by direction the gap is 10.3 points and 9.7 points. The finding survives and gets sharper: the false-alarm-side gap is positive for all eight alternate judges, while the missed-failure side is negative for two of them — so the pooled phrasing was concealing a sign change. Same error class as the confound above, in the section whose subject is conditioning on the wrong thing.
Join-key collision (33% of τ-bench records)
A (task_id, trial, domain) key silently collapsed agent identity: 660 of 1,980 τ-bench records were scored against the wrong agent's reward signal. Detected because one qualitative case contained a state the classifier itself could not produce. The report was wrong. The repo records the correction and the corrected values.
Mathematically unpassable gate
A preregistered task-level threshold (1.25×) was computed only after outcomes were visible. When computed, it was mathematically unreachable: 81.8% of tasks held a failing trial, capping the indicator below the threshold by construction. Reclassified as non-decision-bearing rather than treated as a failure gate.
Oracle / null comparison across incompatible outcome variables
An early draft compared oracle lift to null-rule lift using two different outcome variables, inflating one figure by approximately 6×. That error class — comparing rates across incompatible conditionals — is now at six instances, three of them found externally on 2026-10-02, and each of the first three landed immediately after the prior one was corrected. This instance was caught before publication. The later ones were not.
Claim withdrawn from synthesis: “four of five mechanisms failed”
A synthesis-level summary compressed the heterogeneous mechanism dispositions into a ratio that had no enumerated denominator and converted two "weakened / no binary gate" outcomes into failures. Withdrawn during the final internal-consistency audit; the summary now names dispositions mechanically.
Full process-failure audit — all 24 entries (13 internal, 11 external), their evidence classes, and whether each was caught before or after outcomes were visible: docs/research_synthesis.md §10.
What the study established
Not a framework. A set of distinctions that were non-obvious going in and are hard to recover from offline metrics alone:
- →
Measure the deployed evaluator before designing around hypothetical stochasticity.
Five draws at temperature 0.0 left only 20 of 1,106 cases with any intermediate correctness. The variance to recover was not there. Measurement costs one experiment; a five-copy production system costs ongoing inference spend.
- →
Separate missed failures from false alarms.
Primary errors split 130 false alarms to 32 missed failures. A mechanism that recovers one stratum well may be useless or harmful on the other. Pooled metrics obscure this until you look.
- →
Distinguish reference-conditioned statistics from executable policies.
P(alt correct | primary wrong) is not an operational recovery rate. It conditions on a ground-truth label a runtime policy cannot see. Matched within error direction the gap is 10.3 points on missed failures and 9.7 points on false alarms — stated here as "20–30 points" until 2026-10-02, which came from differencing a pooled rate against two direction-specific ones.
- →
Hold deterministic rules to held-out validation too — and to the gate they were actually given.
R1's pre-check half fired at 0.51× the base rate on held-out data, inverted from its design-corpus behaviour. R4's fired at 1.20×, in the predicted direction. Neither had a numeric accept/reject threshold: §7 of the preregistration gave both of them a direction and no bar, because neither changes a verdict. This page called R4 a failure against a threshold it never had, which is a post-hoc judgment dressed as a preregistered one.
- →
Allow UNVERIFIABLE when required evidence is structurally absent.
R3 does not claim a verdict. It claims the evaluator lacked evidence, which is the right claim to make when it is true and the easiest kind to validate. This entry used to cite R3 as the mechanism with the cleanest pass — it was the one rule that passed its numeric gate cleanly, but not the only rule accepted: R2 was accepted too, underpowered. The pass itself stands — the preregistered test ran as specified and met its criterion — but its reading was narrowed on 2026-10-02, so the argument now rests on the structure of the claim rather than on the size of the separation. Narrowed, not withdrawn: the within-slice residual stays positive and in the predicted direction.
- →
Condition every pooled contrast on the benchmark it was pooled over.
R3's separation was +11.2 pp pooled and +5.8 pp once benchmark identity was held fixed, because all 133 of its firings landed in the one slice carrying the highest judge-error point estimate (22.4%, against 19.8% on the next-highest, 11.1% and 3.9% on the other two). Any rule that fires preferentially on the cases a judge finds hard passes a pooled error-rate test whether or not it carries information. That objection was written down — not in the preregistration, which is where this page credited it until 2026-10-02, but in a caveat added to the results write-up after R3's numbers existed: "Any rule that escalates the cases a judge finds hard will pass a test of the form 'is the judge worse on the escalated subset'. That test is necessary, not sufficient, and the preregistration should have said so." It then published ACCEPTED two paragraphs later without running the check. R3 was also not the only escalation signal available: R1's escalation half separated 33.3% against 15.2% (9/27 vs 187/1,232, Fisher p = 0.026) across three slices rather than one — post hoc, with fired arms of 7, 10 and 10 cases against unfired arms of 283, 363 and 458, and reported nowhere until 2026-10-02. At those fired n it is not obviously confounded, which is weaker than not confounded; this sentence said "every within-slice arm under n = 20" until 2026-10-02, and the comparison arms are not small at all.
- →
Use cheap baselines before crediting semantic machinery.
A null rule requiring no obligation model reached 1.171× where RC1 reached 1.101×, but paired on the same task resamples that difference is +0.078 [−0.327, +0.575] — the two are not separated, and this page previously stated the ordering as if they were. (+0.078 is the mean of the paired resample differences; differencing the two point estimates above gives +0.070. The interval straddles zero either way. This entry rendered the pair as "+0.08 [−0.36, +0.58]" until 2026-10-02 — rounded, and the interval was wrong in both bounds.) A semantic detector should clear a null floor before being given authority; RC1 did not clear it and did not fall below it either.
- →
Preserve correction history.
7 of the 13 errors this study found about itself were caught after the outcomes they affected were visible, and a further 11 were found by a separate reproduction run from outside the project, on a frozen copy. None of the 11 required new data — they were all recoverable from the published numbers by someone willing to check the arithmetic and ask what each conditional was conditioned on. In a study about catching errors in evaluations, the process errors are part of the result.
What I would build differently now
- –Instrument evaluator inputs and provenance before adding more judges — know what information the evaluator is operating on before deciding whether to supplement it.
- –Make missing evidence an explicit state, not a tie-broken verdict.
- –Define false-alarm and missed-failure budgets separately before choosing a mechanism.
- –Evaluate production-deployable decision rules rather than oracle-conditioned recovery statistics.
- –Preregister important evaluation comparisons, including the choice of comparison metric.
- –Make join-key and cardinality assertions part of every benchmark pipeline.
- –Always include a cheap or null baseline before crediting semantic machinery with the lift.
- –Report where every rule fires before reporting what it separates — a firing distribution across benchmarks is one line of code and would have caught the R3 confound before it was published.
- –Make every hazard the project has written down anywhere — preregistration, results caveat, review note — into an assertion the analysis runs, not a sentence a reader is trusted to remember from another file. The R3 hazard was in a post-results caveat, which is the worst place for it: late enough to not gate anything, early enough to prove it was seen.
Methodology and audit trail
I started this work expecting to justify a richer evaluator-control architecture. The useful result was narrower: most of the mechanisms I tested did not earn the authority I wanted to give them. The repo preserves the failed gates and corrections because those are part of the result.
Published 2026-09-29 · corrected 2026-10-02 · Four-day research sprint · Two public corpora · No synthetic trajectories · Back to corpus