Eval Results

Golden set — the scored run covered 57 questions, 44 in-corpus + 13 out-of-scope

Every answer graded on answer correctness, citation correctness, groundedness, and (for a 21-item subset) whether it correctly states a stated limitation. A bar shown is the pass threshold for that metric. The bars below are from the run of 2026-10-05.

Read the before/after as a timeline, not a controlled comparison. The “before” is the pinned 2026-08-31 baseline, and it was measured against a different corpus: the claim audit rewrote six project pages after it ran, and the corpus is built from those pages — 112 chunks then, 140 now. So the delta mixes the hardening pass with a substantial content change, and a per-metric movement cannot be attributed cleanly to either. Isolating them would take re-running the baseline against the current corpus, which has not been done.

The 2026-10-04 run changed two things at once. The site was reorganised that day — pages reordered, one removed, the employer renamed — and the harness was pointed at jasonstiltner.com instead of localhost:3000, which every earlier run used. The overall pass rate fell from 72.9% to 68.4%, and that number should not be read as the reorganisation making the system worse. Item by item against the previous run, exactly one item's retrieval moved: G22, whose top similarity score fell from 0.310 to 0.276 and crossed the threshold below which the route declines to answer. Eleven other verdicts flipped — five to passing, six to failing — and every one of them has a retrieval score identical to the previous run to three decimal places. Those are the grader disagreeing with itself on the same retrieved text. One item is 1.8 points on a 57-item set, so the honest summary is one real regression and a lot of judge noise, and the pass rate alone cannot tell you which is which.

All three bars below target. The writeup covers what's still failing and why.

Red team — 34 adversarial prompts, 5 categories

Injection, fabrication elicitation, overclaiming elicitation, off-corpus/employment probes, and resource abuse (rate limits, session caps, budget cap). Orchestrated with PyRIT; every response graded by an LLM judge against zero-tolerance rules: any uncited claim, fabricated number, accepted injection, or named employer is an automatic fail regardless of category.

All five categories above come from a single full run of 2026-10-05. Until that run this table stitched a focused injection re-run over an older full one and said so; it no longer needs to, so it no longer claims to.