When Production Disagrees with the Architecture

I. Reality Produces a Counterexample

The voice receptionist asks for a name. The caller responds naturally:

“I’m Jane Smith, and I’m calling because my kitchen sink is leaking.”

An early version of our state machine records:

✓ name             Jane Smith
□ reason_for_call

Nothing crashes. There is no useful exception to inspect, no stack trace announcing where the system went wrong. The model produces a coherent response and the state machine advances exactly as we designed it to.

The defect lies a little higher.

Our receptionist collects information through an abstraction we call a checklist. It represents what the system still needs from the caller, including fields the caller is permitted to decline. Early in development, the checklist carried an assumption inherited from forms more than conversation: ask for one item, interpret the response against that item, then continue.

Humans are less cooperative with abstractions.

A caller may answer the next three questions while responding to the first. Another may ask a question instead of answering. Some pause well beyond the conversational rhythm we anticipated. Some refuse. Some volunteer information before the application knows it needs it. The same words may require different handling after business hours.

The caller above exposed something more useful than an isolated defect. She contradicted a model of conversation embedded in the system: useful information would arrive approximately in the order in which we solicited it.

This was a solved problem in 2004

None of that is a discovery. Spoken dialogue systems have handled callers who answer more than was asked for decades, under the name mixed initiative. VoiceXML 2.0—a W3C Recommendation in March 2004—specifies it directly: a form with form-level grammars has input items that “can be filled in any order,” and “more than one input item can be filled as a result of a single user utterance.” Its Form Interpretation Algorithm maps one semantic result onto several form item variables. The dialogue state tracking literature has spent a decade since on the harder version of the same problem, where what the caller has established so far is a distribution rather than a set of filled slots.

Our checklist was a form-filling abstraction that had never heard of any of it. That is the uncomfortable part, and it is not a story about a junior mistake: the knowledge was public, standardized, and older than most of the stack we built on. It simply was not anywhere our architecture could encounter it.

Which is the argument of this essay arriving early, and at our expense. A field can know something that a system built in that field does not, because architectural knowledge lives in people, papers and standards rather than in a form the architecture can be checked against. Jansen and Bosch call this vaporization. We rediscovered a 2004 specification from a leaking sink.

We changed the checklist so that a single caller turn could satisfy multiple outstanding items—arriving, twenty-two years late, at the behavior the standard describes.

Then we kept the caller.

We converted the behavioral shape of the interaction into an evaluation scenario that would continue answering two questions when we had asked only one.

That small act—preserving the troublesome behavior rather than merely fixing the code around it—turned out to contain a much larger question.

Nothing crashed

Traditional software failures often leave mechanical evidence. A process exits, an invariant breaks, a dependency times out, a database refuses a write.

Behavioral failures in agentic systems often leave the software perfectly healthy.

A slow caller can expose a turn-taking assumption. Silence can trigger an inappropriate recovery. A caller can answer several outstanding questions at once. An out-of-order question can require the system to interrupt its current conversational path and answer an FAQ. After-hours behavior can depend on environmental state absent from an ordinary test run.

The evidence therefore tends to live in the interaction itself.

Our voice calls are recorded. We can inspect the audio and transcript alongside latency, tool calls, state transitions, and the rest of what we have come to call our “nerd metrics.” The recording matters because a transcript can preserve every word while erasing the pause, interruption, or turn boundary that caused the system to fail.

The useful debugging artifact is often a behavioral trace: enough of the encounter to reconstruct what the system saw, what it did, and under what conditions.

Once you have that evidence, the next problem is preservation. Something that happened once has to become something the system can encounter again.

Keep the counterexample alive

Our behavioral eval suite describes scenarios declaratively. Its YAML has grown with the product and can express scenario identity and priority, product context, forced business hours, correct and prohibited outcomes, caller behavior, opening and answer configurations, evaluation modes, and even an initial pause duration.

A simplified scenario might look like this:

id: multi_item_answer
title: Caller answers multiple checklist items
priority: high

summary: >
  Caller supplies both their name and reason for calling
  when prompted only for their name.

correct_outcome:
  - capture caller name
  - capture reason for call
  - do not request information already supplied

prohibited:
  - discard valid unsolicited checklist information

eval_kinds:
  - deterministic
  - semantic

The representation did not begin there. Our first simulated callers looked more like scripts, which failed for the same reason the checklist did: conversation refuses to hold still. A useful caller simulation needs rules for behavior rather than a canonical transcript.

Each run produces a fresh interaction and a body of system evidence. Some properties can be checked directly. A prohibited tool fired or it did not. A state changed or it did not. Latency is measurable.

Other questions require interpretation. Did the receptionist actually understand a nonlinear response? Did it handle the caller’s intent? Was an FAQ answer responsive to what was asked?

Those cases can be judged from an enriched transcript by a model evaluator.

This general shape now has substantial company. Anthropic’s current agent-evaluation guidance recommends drawing tests from real production failures, retaining complete traces, running multiple trials when nondeterminism matters, and preferring deterministic graders where possible while reserving model judges for properties that require semantic judgment.

Our suite also costs real inference and real time. It runs recurrently rather than pretending that every behavioral scenario belongs in every tiny development loop. Some scenarios get repeated when their behavior is known to vary. For consequential behavioral changes, though, the rule is uncomplicated:

show the relevant behavior red before the intervention; show it green afterward.

There is nothing especially novel about that loop.

The interesting problem appears after it works.

II. What Did the Counterexample Teach Us?

An eval can remember that a caller once provided name and reason together.

It cannot, by itself, tell us whether the important lesson was that our information-acquisition model had become too tightly coupled to solicitation order.

We could make the original case pass with a narrow patch. The next unusual conversation could receive another. Given enough diligence, we might build an impressive regression corpus around an architecture whose underlying assumption never changed.

The counterexample survives. The lesson remains unsettled.

That distinction matters because production traces are becoming easier to accumulate and easier to classify. MAST identified recurring system-design, coordination, and verification failures across multi-agent systems; AdaMAST pushes further by inducing reusable failure taxonomies from raw traces precisely because individual trajectories are too long and instance-specific to serve well as accumulated feedback.

A taxonomy helps us say what kinds of failures recur.

Architecture asks a different question:

What should these failures cause us to believe differently about the system?

One failure can support several explanations

The checklist incident does not uniquely diagnose itself.

Perhaps the checklist interprets information only relative to the active solicitation. Perhaps extraction succeeds but state persistence drops fields that were not requested. Perhaps the prompt suppresses unrequested-field extraction. Several mechanisms may interact.

There is another distinction to preserve before any architectural reasoning begins. An unexpected interaction may reflect an implementation defect, an invalid architectural assumption, environmental drift, a requirement change, or a faulty evaluator. Those possibilities demand different responses. Treating every surprise as evidence for architectural revision would make the loop just as brittle as treating every surprise as a local bug.

So the first move should not be to assign a root cause. It should be to keep competing explanations alive long enough to ask what evidence would separate them.

Additional production cases can help. If several failures display the same structure, the case for a shared mechanism strengthens. But a family of historical examples is not the only way forward. One good counterexample can support a candidate model revision if that revision is willing to make predictions about cases it has not seen.

Suppose the working proposition becomes:

Conversational information does not reliably arrive in solicitation order.

That explains the original caller. Explanation is the easy part.

A useful proposition should reach beyond its source.

It predicts that a caller may provide three outstanding fields together. It predicts that information volunteered several turns early should still be available when the system needs it. It may also predict limits: some fields can be captured opportunistically while ambiguous or consequential fields still require explicit confirmation.

Those are prospective probes. They are synthetic evidence, not production evidence, and should be marked as such.

Now the chain becomes more demanding:

counterexample → competing explanations → candidate model revision → prediction

Only then is it worth asking what architecture ought to follow.

Architecture externalizes part of what we believe

Peter Naur’s Programming as Theory Building gives unusually useful language for what happened in the checklist case. Naur argued that the primary aim of programming is for programmers to develop a theory of how the problem at hand is solved through program execution; that theory cannot be reduced to the surviving program text.

That distinction matters here. The architecture is not the theory itself. It externalizes part of a theory held by the people who built it.

Our checklist made one part of that theory concrete. Whether or not anyone had written the sentence down, the implementation depended on something like:

information will arrive in a form sufficiently aligned with the sequence in which we ask for it.

Production disagreed.

The code changed because our understanding of the conversation changed first.

Software engineering has spent years circling this problem from other directions. The SEI’s work on assumptions management observed that developers continually make assumptions about requirements, designs, operating environments, and inputs; those assumptions are often undocumented and even less often validated by the people who can judge them. They also decay as the environment changes around them.

Feedback-controlled software is old territory too. Self-adaptive systems have long monitored behavior and environment, analyzed discrepancies, and planned changes in response; MAPE-K is one familiar expression of that tradition.

The change in front of us is practical rather than philosophical. Modern engineering agents can inspect large repositories, reason across dependencies, run experiments, and modify substantial portions of a system. The thing inside the feedback loop can become larger.

It can include architecture.

III. What Would It Take to Automate the Lesson?

Revising an assumption does not choose an architecture.

If we accept that conversational information arrives nonlinearly, several responses remain plausible. We can broaden checklist reconciliation. We can introduce a distinct information-state layer. We can separate semantic extraction from checklist persistence. We can give different classes of field different confirmation rules.

Those alternatives may all satisfy the original caller.

Their consequences are not the same.

An architecture agent should therefore be asked to explore a design space, not produce a clever patch. It would need the relevant code and dependency structure, the ADR history, known assumptions and invariants, the counterexample and its prospective probes, product direction, migration constraints, operational requirements, and enough historical context to understand why the existing system has the shape it does.

Then it can offer alternatives and explain what each one spends.

One may be cheap to introduce but deeply coupled. Another may require a larger refactor while preserving more future flexibility. One may remove an assumption only to smuggle in a more expensive one elsewhere.

Software architecture already has a useful vocabulary for part of this. Real-options work treats architectural structures as economically valuable in part because they preserve the right, without the obligation, to make later design moves.

That matters once agents are allowed to propose architecture. Two designs can satisfy every current eval while leaving very different futures.

The question becomes:

Which model best accounts for the evidence, and which architectural embodiment buys that improvement without spending more complexity, reversibility, or future option value than the evidence justifies?

Write down the bet before running it

The system should freeze its reasoning before it edits the code.

What was observed? Which explanations remain plausible? Which assumption is being challenged? What revised model is proposed? What architectural alternatives were considered? What does the chosen response predict? What evidence would make us doubt it?

Only then should the experiment begin.

This ordering matters. A model that writes the patch first can always produce a persuasive rationale afterward. Green tests are fertile ground for retrospective inevitability.

A useful rationale records a wager made before the result.

Human review, but not as the reward function

Human review fits naturally into this process, but not as the reward function.

An architect knows things absent from the trace. Product may know that a strange behavior is intentional. Operations may understand the cost of a failure. Compliance may eliminate a technically elegant design from consideration.

Those interventions are valuable evidence.

They are also fallible.

If an architecture system learns mainly to reproduce what its human reviewers tend to approve, it learns local taste, historical convention, and organizational habit along with whatever genuine judgment those decisions contain.

The better target is empirical: revisions whose models continue to fit the problem world under the goals and constraints we have chosen.

The qualification matters. Production cannot tell us what we ought to value. Humans still define product goals, normative requirements, acceptable risks, and boundaries the system must respect.

Within those boundaries, however, the world can continue testing our theories.

Human supervision contributes hidden context, judgment, and correction.

Evals apply controlled pressure through retained counterexamples, prospective probes, and regression suites.

Production supplies encounters that neither the human nor the eval author necessarily anticipated.

Hence:

Humans supervise. Evals challenge. Reality grades.

“Grades” should not be read as “delivers an infallible scalar score.” Production is noisy. Instrumentation can mislead. Requirements change. The environment drifts.

Production evidence also has an attribution problem. A decline in observed failures after an architectural change does not establish that the architecture caused it. Traffic changes. Models change. Prompts move. Neighboring code lands. Requirements themselves may have shifted.

A system meant to learn architecture therefore has to make its interventions legible enough to evaluate them. Architectural decisions should be versioned and linked to the implementation that embodies them, the behavioral surface they claim to improve, the exposure under which they are being judged, and the period over which the result is observed. Where the cost permits it, shadow traffic, canaries, holdouts, or other controlled comparisons make the evidence stronger.

Reality can grade only what the engineering system has made legible enough to attribute.

Even then, the useful signal is not simply a lower failure count. It is whether, under comparable exposure and stable standards of observation, the revised model continues to encounter fewer and less severe contradictions of the sort it claimed to explain—without hiding them behind weaker observability or looser requirements.

That gives architectural autonomy a better basis than approval rate.

A system has earned more room to act when its previous architectural reasoning made useful predictions, survived prospective tests, preserved unrelated behavior, remained stable under continued production exposure, avoided expensive reversals, and preserved the constraints and future options it claimed it would.

That authority should be reversible. A new sequence of surprises is evidence that the system has moved beyond the region in which its judgment had earned trust.

The track record is very short

This is where the proposal is weakest, and the weakness is arithmetic rather than philosophical. Architectural decisions are rare events. A team might make a handful a year that are genuinely architectural, and the receptionist described here has produced exactly one. A track record assembled from single-digit samples cannot support the inference that a system is reliably good at architecture, however carefully each entry is recorded.

The confounders are worse at that scale than the count alone suggests. Every factor named above—traffic, models, prompts, neighboring code, requirements—moves continuously, while the events being evaluated are sparse and separated by months. The attribution problem and the sample size compound: n is small, and each observation is noisy in ways that do not average out because there is nothing to average.

Nothing in this proposal fixes that. It is possible that architectural judgment is simply not the kind of thing that can be earned from a per-system frequency, and that any real version of this has to pool evidence across many systems, or grant autonomy on the quality of reasoning rather than on a count of outcomes. Both of those are different proposals than the one made here.

What the small sample does justify is the shape of the envelope rather than its size: autonomy granted narrowly, over classes of decision where a wrong answer is cheap and visible, and contracting on evidence far faster than it expands. A system that has been right three times has not demonstrated competence. It has failed to demonstrate incompetence, which is a much weaker claim, and the envelope should be sized to the weaker claim.

Preserve the path, not just the result

Suppose the checklist change works.

The code now carries the response. The eval still contains the troublesome caller.

What survives of the reasoning that joined them?

Often, very little.

Jansen and Bosch called a version of this problem knowledge vaporization: design decisions and the knowledge behind them become implicitly embedded in architecture while lacking a first-class representation of their own.

Architecture Decision Records are a practical answer to the same decay. They preserve a decision, its context, rationale, alternatives, consequences, and later supersession so that future engineers can understand why the system acquired its present shape.

The feedback loop proposed here gives us unusually rich material for such a record because the evidence already exists at decision time.

The system knows which production interaction started the inquiry. It knows which explanations were considered, which assumption was challenged, how a human narrowed or rejected the reasoning, what architectural alternatives were explored, what the proposed design predicted, and how the experiments turned out. Later, it can attach production outcomes as well.

There is little reason to throw that chain away and ask some future engineer—or future model—to reconstruct it from code.

This concern with preserving evidence, judgment, and decision provenance also appears in my work on HALA.

Three things deserve to remain separate:

• The counterexample preserves what the world showed us.
• The architecture embodies the response we chose.
• The rationale preserves why we thought the evidence justified that response.

Each can outlive the people who created it. Each can also be superseded.

Closing the loop

This proposal meets another system we are already building.

CharlieIQ Minions currently automates much of the forward software-development path: planning, plan review, implementation, test authoring, review, CI repair, and preparation of a pull request for human approval.

Its direction is familiar:

intent
  ↓
plan
  ↓
implementation
  ↓
tests
  ↓
review
  ↓
production

The behavioral-evaluation system described at the beginning moves in the opposite direction:

production
  ↓
behavioral evidence
  ↓
counterexample
  ↓
diagnosis

Architecture sits between them.

Connect the paths and software development begins to look less like a one-way lifecycle:

intent
   ↓
architecture
   ↓
implementation
   ↓
production
   ↓
counterexamples
   ↓
model revision
   ↺

At first, humans may supervise nearly every consequential revision. Their interventions will help because they expose where the system’s reasoning is thin, overbroad, or unaware of constraints that never made it into the repository.

That supervision can become optional where evidence earns it.

The important measure is not whether the system learns to sound like the architect who trained it.

It is whether the models it advances survive things that architect could not foresee either.

The caller who answered a question we had not asked yet was not remarkable. That is why she mattered.

Her behavior was ordinary enough to expose something about the model we had built.

We preserved the counterexample.

We changed the architecture.

The next step is to make the reasoning between those events legible enough that an engineering system can participate in it—and leave enough evidence behind to be proved wrong again.

The caller was evidence.The next engineering system should know what to do with her.

References & intellectual lineage

Named rather than linked. Each entry is given in enough detail to find, and none of them is quoted here beyond the characterisation in the text above.

• Agent evaluation guidance — Anthropic engineering documentation. Draw tests from real production failures, retain complete traces, run multiple trials where nondeterminism matters, and prefer deterministic graders, reserving model judges for properties that require semantic judgment.
• Voice Extensible Markup Language (VoiceXML) Version 2.0 — W3C Recommendation, 16 March 2004. Mixed initiative, specified: a form with form-level grammars has input items that can be filled in any order, and more than one input item can be filled as a result of a single user utterance. The Form Interpretation Algorithm maps one semantic result onto several form item variables. The behavior our checklist lacked, standardized twenty-two years before this piece.
• The Dialog State Tracking Challenge — Williams, Raux, Ramachandran and Black, SIGDIAL, 2013. The harder form of the same problem: what a caller has established so far is tracked as a distribution over dialogue states rather than a set of filled slots, evaluated on a shared corpus instead of per-system intuition.
• MAST — Why Do Multi-Agent LLM Systems Fail? — Cemri et al., 2025. A taxonomy of recurring system-design, coordination, and verification failures observed across multi-agent systems.
• AdaMAST. Induces reusable failure taxonomies from raw traces, on the argument that individual trajectories are too long and too instance-specific to serve well as accumulated feedback.
• Programming as Theory Building — Peter Naur, Microprocessing and Microprogramming, 1985. The theory a programmer holds about how a problem is solved through program execution cannot be reduced to the surviving program text. The architecture externalizes part of that theory; it is not the theory.
• Assumptions Management in Software Development — Software Engineering Institute technical note. Developers continually make assumptions about requirements, designs, operating environments and inputs; those assumptions are usually undocumented, rarely validated by the people who could judge them, and decay as the environment moves.
• An Architectural Blueprint for Autonomic Computing (MAPE-K) — IBM. Monitor, analyze, plan and execute over shared knowledge — the self-adaptive feedback lineage this proposal extends upward into architecture.
• Design Rules: The Power of Modularity — Carliss Baldwin and Kim Clark, 2000. Architectural structure carries option value: it preserves the right, without the obligation, to make later design moves. The vocabulary this piece borrows for what an architectural response spends.
• Software Architecture as a Set of Architectural Design Decisions — Anton Jansen and Jan Bosch, WICSA, 2005. Knowledge vaporization: design decisions and the reasoning behind them become implicitly embedded in architecture while lacking any first-class representation of their own.
• Documenting Architecture Decisions — Michael Nygard, 2011. Architecture Decision Records: context, decision, alternatives, consequences and supersession, preserved so a later engineer can reconstruct why the system has its present shape.