# Process artifacts

Supporting files for [/writing/agentic-delivery/](https://jasonstiltner.com/writing/agentic-delivery/),
which describes how this site's content was produced and verified. Every number in that essay
should be checkable against a file in this directory.

## Contents

| File | What it is | Form |
|---|---|---|
| [spec-and-amendments.md](https://jasonstiltner.com/process/spec-and-amendments.md) | The brief, its two amendments, the supersession rule, and every correction the agent made to its own instructions | Redacted excerpt |
| [open-questions.md](https://jasonstiltner.com/process/open-questions.md) | Where the brief asked for something the fact inventory didn't support, and the agent wrote the question down instead | Redacted excerpt |
| [evals/README.md](https://jasonstiltner.com/process/evals/README.md) | The eval workspace for "Chat with the Research": phase log, pass-rate bars, before/after data, and the findings that are still open | Verbatim copy |
| [evals/golden_set.md](https://jasonstiltner.com/process/evals/golden_set.md) · [.json](https://jasonstiltner.com/process/evals/golden_set.json) | 50 graded items — 35 in-corpus question/answer pairs, 15 out-of-scope prompts that must be refused | Verbatim copy |
| [evals/adversarial_taxonomy.md](https://jasonstiltner.com/process/evals/adversarial_taxonomy.md) · [.json](https://jasonstiltner.com/process/evals/adversarial_taxonomy.json) | 34 red-team prompts across 5 attack categories | Verbatim copy |
| [evals/results/](https://jasonstiltner.com/process/evals/results/golden_run_2026-08-31.json) | Six dated harness runs, including every per-item model response | Verbatim copy |
| [evals/harness/](https://jasonstiltner.com/process/evals/harness/golden_harness.py) | The six Python files that produce those runs — the graders, the judge, and the endpoint client | Verbatim copy |

### The six runs

- `golden_run_2026-08-31.json` — first real baseline
- `golden_run_2026-08-31-after-phase4.json` — after the first hardening pass
- `golden_run_2026-08-31-after-phase4-chunking.json` — after the second
- `redteam_run_2026-08-31.json` — red-team baseline
- `redteam_run_2026-08-31-after-phase4.json` — after the first hardening pass
- `redteam_run_2026-08-31-t105-recheck.json` — recheck of a single injection finding

### The six harness files

There is no directory index, so these are listed rather than browsable:

- [`golden_harness.py`](https://jasonstiltner.com/process/evals/harness/golden_harness.py) — runs the 50 golden items and scores them
- [`redteam_harness.py`](https://jasonstiltner.com/process/evals/harness/redteam_harness.py) — runs the 34 adversarial prompts
- [`judge.py`](https://jasonstiltner.com/process/evals/harness/judge.py) — the model-graded rubric; note that the judge is itself a model, which the essay names as a limitation
- [`endpoint_client.py`](https://jasonstiltner.com/process/evals/harness/endpoint_client.py) — calls the live `/api/research-chat` route
- [`pyrit_target.py`](https://jasonstiltner.com/process/evals/harness/pyrit_target.py) — PyRIT adapter for the red-team pass
- [`env.py`](https://jasonstiltner.com/process/evals/harness/env.py) — loads API keys from a gitignored `.env.local`

Publishing the harness is safe because keys are read from the environment and never appear as
literals. It is published because a result file a reader cannot regenerate is a claim, not a
receipt.

## On the two redacted files

The source documents are internal review notes for a private repository. They contain three
things that cannot be published: detail that identifies the current employer by description
rather than by name, working notes for a job search, and the method used to probe repository
visibility. Each excerpt states at its head what was removed, and
[open-questions.md](https://jasonstiltner.com/process/open-questions.md) states how many items
were withheld and why, by category.

Nothing was removed because it was unflattering. Every correction the process made against
itself — including the one it made against its own verification record — is in the excerpts.

## How these files are maintained

The `evals/` files are copied byte-for-byte from the source workspace by
`scripts/publish-process-artifacts.ts` on every build, and CI fails if a copy is stale. The two
excerpts are hand-written, because redaction is a judgement no script can make; they are
therefore outside that guarantee, and they carry a date rather than a freshness check.

Last reviewed: 2026-09-03.