Agentic Delivery: Specs, Supersession, and Eval Gates

Why this exists

Agent output is cheap. Knowing what is true after the agent has run is not. A thousand lines arrive in a minute, and the expensive question — which of these claims survives being checked — is exactly the one the speed makes easy to skip. The process below is an attempt to make that question answerable after the fact, by a reader who was not there.

Spec and amendment

The work ran from a base brief of seven sections and two amendments, of seven and six sections. Eighteen commits, one per section. Amendments were issued as documents, not as chat corrections: each one names the sections it supersedes, and a superseded section is not deleted or edited — it stays in the record, marked.

The distinction matters because a chat correction leaves no trace once the context window rolls over. A superseding document leaves a record in which any claim on the live site can be traced back to the instruction that produced it. The note at the head of the Amendment 1 section, verbatim:

spec-and-amendments.md — the section-by-section record.

Supersession is the load-bearing rule

A worked example. Amendment 1 shipped a credentials band on the homepage asserting founding-engineer tooling work. Amendment 2 corrected the underlying fact: third engineer hired, first Staff Engineer — stated plainly rather than softened, and corrected in all four places it appeared. Amendment 1's section was not rewritten to match. It remains in the document, describing what shipped, with the note above at its head.

Without that rule the record of what the agent was told drifts toward the record of what the site currently says, and the two stop being independent. Once they are the same document, no claim can be audited back to its instruction, and the record can no longer contradict the site — which is the only thing it is for.

The fact inventory

One section of the brief was not an instruction to write anything. It was a list of verified facts, and a standing rule: don't assert what isn't covered. The observable consequence is a list of nineteen items the agent flagged rather than filled. Two of them:

• Asked to reflect a work history, it declined to build a dated timeline. The source's own ordering was ambiguous — "Earlier: [employer]" doesn't say earlier than what — and it asked for the real order instead of guessing one.
• Given a version-history example in the brief itself (18 patterns, +6 new), it found the numbers didn't reconcile to the documented total of 28, and derived 20 + 8 from the per-pattern isNew flags in the data. It then recorded that this was not the number the brief suggested.

Six of the nineteen are published; thirteen are withheld, with the count and the reason for each category stated. open-questions.md.

Verification loops

Each section ended with a build and a check. The checks that found things were the mechanical ones: a build that proved a claimed rendering defect did not exist, and greps producing before/after counts for specific strings across the whole repository rather than a spot inspection of the files the agent had just edited.

The most useful result came from a grep for scope language. Amendment 1's own rules forbade job-seeking framing in page copy; a grep found two large sections of it already live on the site, pre-existing, including a subheading reading "Why I'm Seeking Senior ML Engineering Roles." Neither the brief nor the amendment had pointed at that page. The check found it because it was a check over the repository rather than over the diff.

The defect the process found in itself

Amendment 2 required that the current employer not be named on any public surface. The verification section recorded the check as a table, one row per target string. One row covered the employer's name, its parent, and a branded product of theirs, and asserted zero hits everywhere, including git history.

To say so, the row spelled all three strings out. Committing that sentence wrote them into a tracked file and into git history. The claim was false from the moment it was recorded — not because the grep was wrong, but because writing the result down invalidated it. It was found on 2026-09-01, two days after it was committed, while vetting the document for publication.

No published surface was ever affected; src/ and public/ were clean before and after. The document now describes the check without naming its targets, and the history claim is scoped to the commits that preceded it. The history was not rewritten, and the correction says so.

This is the fifth self-correction in the record and the only one made against the verification record itself. The other four are quoted verbatim in spec-and-amendments.md, including one the agent made against a claim it had asserted earlier in the same session.

Eval gates

The site's research-chat feature was gated on bars fixed before any result existed: 95% citation-correctness, 95% correct-refusal, 90% limitation-inclusive answers, and zero tolerance on every red-team category. The graded set is 50 items — 35 in-corpus question/answer pairs and 15 out-of-scope prompts that must be refused — plus 34 red-team prompts across five attack categories.

Two of the four bars are still unmet. Citation-correctness finished at 91.2% against a 95% bar and limitation-inclusive answers at 75.0% against 90%. The feature shipped with those numbers written down rather than with the bars lowered to meet them. One retrieval-recall failure was left deliberately unfixed: the change that would have fixed it degraded other items, and the trade-off is recorded instead of resolved.

The middle column is the reason to publish all six runs rather than the endpoints. The first hardening pass improved the overall score and made refusals perfect while limitation-inclusive answers regressed, 66.7% to 50.0% — a real cost that a before/after summary would have hidden.

evals/README.md · golden set · red-team taxonomy · baseline run · final run. The result files carry every per-item model response, which is the part that makes them a receipt rather than a summary.

Limitations

This is n = 1. One site, one agent, eighteen commits, no control condition. Nothing here establishes that the spec-and-amendment structure caused the outcome rather than accompanying it, and no comparison was run against the same work done without it.

The eval judge is itself a model, and the bars were set by the same person who grades against them. A personal site is also the easy case: the coordination costs that dominate real delivery — several people editing the same spec, disagreement about what a fact is, deadline pressure against a failing bar — are absent by construction. The rule that a verification record must not name what it certifies absent is a general one. The evidence that the surrounding process scales is not here.

Two of the four published documents are redacted excerpts, not full sources. Each states what was removed and by category, and the withheld counts are stated rather than elided — but a reader is trusting that the redaction removed only what it says it removed.

Artifacts

• /process/README.md — index and redaction policy
• spec-and-amendments.md — the brief, both amendments, the supersession rule, every self-correction (redacted excerpt)
• open-questions.md — six of nineteen flagged items, and the categories of the thirteen withheld (redacted excerpt)
• evals/ — golden set, red-team taxonomy, and six dated runs with per-item responses (verbatim copies)

The eval files are copied byte-for-byte from the source workspace on every build and CI fails if a copy is stale. The two excerpts are hand-written, because redaction is a judgement no script can make; they are outside that guarantee and carry a review date instead. The source repository is private.