> **Eval Results** > Committed pass-rate data for the Chat with the Research tool: a 50-item golden set and a 34-prompt adversarial red-team suite, scored before and after a hardening pass. > > Source: https://jasonstiltner.com/tools/research-chat/evals/ --- # Eval Results The numbers below are read from [committed harness output](https://jasonstiltner.com/process/evals/README.md) — the same JSON a reviewer could regenerate by running [golden\_harness.py](https://jasonstiltner.com/process/evals/harness/golden_harness.py) and [redteam\_harness.py](https://jasonstiltner.com/process/evals/harness/redteam_harness.py) ([judge.py](https://jasonstiltner.com/process/evals/harness/judge.py)) against this endpoint. What each bar actually means, and the bugs the numbers surfaced, are in the [engineering writeup](https://jasonstiltner.com/projects/chat-with-the-research/). ## Golden set — the scored run covered 59 questions, 45 in-corpus + 14 out-of-scope Every answer graded on answer correctness, citation correctness, groundedness, and (for a 21\-item subset) whether it correctly states a stated limitation. A bar shown is the pass threshold for that metric. The bars below are from the run of 2026-09-14. **Read the before/after as a timeline, not a controlled comparison.** The “before” is the pinned 2026-08-31 baseline, and it was measured against a different corpus: the claim audit rewrote six project pages after it ran, and the corpus is built from those pages — 112 chunks then, 137 now. So the delta mixes the hardening pass with a substantial content change, and a per-metric movement cannot be attributed cleanly to either. Isolating them would take re-running the baseline against the current corpus, which has not been done. Citation correctness · 88.2% → 82.2% (bar: 95%) Correct refusal rate (out-of-scope questions) · 93.3% → 92.9% (bar: 95%) Limitation-inclusive answer correctness · 66.7% → 61.9% (bar: 90%) Overall pass rate · 70% → 72.9% All three bars below target. The writeup covers what's still failing and why. ## Red team — 34 adversarial prompts, 5 categories Injection, fabrication elicitation, overclaiming elicitation, off-corpus/employment probes, and resource abuse (rate limits, session caps, budget cap). Orchestrated with PyRIT; every response graded by an LLM judge against zero-tolerance rules: any uncited claim, fabricated number, accepted injection, or named employer is an automatic fail regardless of category. | Category | Before | After | | --- | --- | --- | | Injection | 6/7 | 7/7 | | Fabrication elicitation | 6/6 | 6/6 | | Overclaiming elicitation | 7/7 | 7/7 | | Off-corpus / employment probes | 7/7 | 9/10 | | Resource abuse | 4/6 | 6/6 | | Overall | 90.9% | 97.2% | All five categories above come from a single full run of 2026-09-14. Until that run this table stitched a focused injection re-run over an older full one and said so; it no longer needs to, so it no longer claims to. **Zero employer-name leaks**, before and after, across every direct and indirect attempt to name Jason's current employer. The rule is enforced in the system prompt and again by an output-side filter on the model's own tokens. [← Back to the chat](https://jasonstiltner.com/tools/research-chat/) · [Read the engineering writeup](https://jasonstiltner.com/projects/chat-with-the-research/)