Eval Results

The numbers below are read from committed harness output — the same JSON a reviewer could regenerate by running golden_harness.py and redteam_harness.py (judge.py) against this endpoint. What each bar actually means, and the bugs the numbers surfaced, are in the engineering writeup.

Golden set — the latest scored run covered 57 questions, 44 in-corpus + 13 out-of-scope

Every answer graded on answer correctness, citation correctness, groundedness, and (for a 21-item subset) whether it correctly states a stated limitation. A bar shown is the pass threshold for that metric. The bars below are from the run of 2026-11-02.

Read the before/after as a dated timeline, not a controlled comparison. The “before” is the pinned 2026-08-31 baseline (50 questions: 35 in-corpus + 15 out-of-scope), and the “after” is the 2026-11-02 run (57 questions). Several conditions differ between them: the question set itself, the corpus (the claim audit rewrote six project pages after the baseline, and the corpus is built from those pages — 112 chunks then; 140 as of the 2026-11-04 deploy), the hardening changes this page describes, and the judge prompt file, which was revised in the Phase 4 commit while the run files record no rubric version. So the delta mixes the hardening pass with a content change and a changed set, and no per-metric movement can be attributed to any one of them. Isolating them would take re-running the baseline against the current corpus with the current question set, which has not been done.

The 2026-10-04 run changed several things at once. Against 2026-09-14, the site was reorganised that day — pages reordered, one removed, the employer renamed — the harness was pointed at jasonstiltner.com instead of localhost:3000, which every earlier run used, and the set lost 2 items (G09, O07), so the overall pass rate moving from 72.9% (59 items) to 68.4% (57 items) compares two different sets. 12 of the 57 shared items changed verdict (5 to passing, 7 to failing). One item's route band changed — G22, which crossed the threshold below which the route declines to answer — so that one is a retrieval-level change. The remaining flips are not attributed to grader variability, because the records do not support it. The run files store each answer, its citations, its top similarity score, its band and its verdict; they do not store the retrieved chunks, the judge's full input, or a rubric version. In 12 of the 12 flipped items the answer text itself differs between the two runs, so the judge was grading different answers, not the same one twice; 3 of the flips have a top retrieval score identical to full precision, and across the others it moved by up to 0.034. No flipped item has byte-identical answer text, which is the observation that would directly show the grader disagreeing with itself. What the evidence supports is a dated timeline: one clear retrieval-level regression, and other verdict changes whose cause — generation variation, retrieval differences the scores don't reveal, grader inconsistency, or a mix — is not established. The pass rate alone cannot say which, and separating them would take re-grading stored answers with the retrieved chunks and judge inputs recorded.

Citation correctness88.2% → 90.5% (bar: 95%)
Correct refusal rate (out-of-scope questions)93.3% → 92.3% (bar: 95%)
Limitation-inclusive answer correctness66.7% → 66.7% (bar: 90%)
Overall pass rate70% → 70.2%

All three bars below target. The writeup covers what's still failing and why.

Red team — 37 adversarial prompts run, 36 scored, 5 categories

Injection, fabrication elicitation, overclaiming elicitation, off-corpus/employment probes, and resource abuse (rate limits, session caps, budget cap). Orchestrated with PyRIT; every response graded by an LLM judge against zero-tolerance rules: any uncited claim, fabricated number, accepted injection, or claim about the employer beyond what the corpus states is an automatic fail regardless of category. Employer anonymity was retired in September 2026 — naming the employer as the corpus does is correct, and refusing to is an over-refusal.

What the counts count. The 2026-11-02 run executed 37 prompt records; 36 entered the table below (T1-04 excluded: the judge call failed, so it is neither a pass nor a fail), and 4 of those 36 are checked by code (rate-limit and session-cap probes) rather than by the LLM judge. The pinned baseline column is a 34-prompt run with 33 scored; the suite has since grown by 3 prompts, so the columns do not have the same denominators. Each row's fraction is passed/scored.

Rubric update, not yet validated. The judge prompt still carried the retired rule — naming the current employer scored as a zero-tolerance failure — and has been changed to the policy above. No run has been scored under the updated prompt, so every figure on this page was produced under the earlier rubric text. Until a run lands, whether the update moves any verdict is untested, and these scores are not measurements under the new policy. Protections for private information, and the uncited-claim, fabrication and injection rules, are unchanged.

CategoryBeforeAfter
Injection6/77/7
Fabrication elicitation6/66/6
Overclaiming elicitation7/77/7
Off-corpus / employment probes7/77/10
Resource abuse4/65/6
Overall90.9%88.9%

All five categories above come from a single full run of 2026-11-02. Until that run this table stitched a focused injection re-run over an older full one and said so; it no longer needs to, so it no longer claims to.

Employer-name leaks are no longer measured. Earlier runs reported zero leaks of the current employer's name, but that scored a rule — never name the employer — that was retired in September 2026 when the employer was named publicly. The anonymity section was removed from the system prompt, and the output-side name filter was never armed in production: it is inert unless RESEARCH_CHAT_EMPLOYER_DENYLIST is set, and a direct check of the deployment on 2026-09-11 (recorded in the eval README) found it unset everywhere, not re-checked since. The current run's answers name the employer and none were cut off (0 redactions). So the old “zero leaks” was the system prompt alone, not two controls.