Reproducible debugging and harness engineering

Context

Difficult production issues — flaky AI features, unreliable webhooks, race conditions, retry storms — share a property: the expensive part is getting the failure to stand still.

This is a method I use, and a design I want more teams to treat as infrastructure. It is also the interface that makes AI-assisted engineering useful instead of theatrical.

Problem

The first occurrence of a failure requires investigation: logs, hypotheses, bisects, local reproduction. That work is inherently expensive.

Teams often stop there. The next similar failure starts from zero. Coding agents then wander through the same log volume, invent the same wrong hypotheses, and open the same incomplete patches.

The missing artifact is a harness: a deterministic way to replay the failure, assert the broken behavior, and keep that knowledge after the patch lands.

Constraints

Production data may be sensitive. Captures have to be redacted or synthesized.

The harness must run in CI, not only on one engineer’s laptop.

It has to be cheap enough that people actually write the second test.

Agents should be able to execute the harness without a guided tour of tribal knowledge.

Architecture

Production failure → capture (inputs, clocks, message order, model/tool traces as appropriate) → reproduction harness → investigation (human or agent) → patch → tests that encode the failure → regression knowledge that remains in the repo.

Engineering decisions

Capture at the boundary where non-determinism enters: time, network, queues, model responses, RNG. Replay inside that boundary.

Prefer a failing test over a narrative bug ticket. Tickets rot. Tests remain executable.

Give agents the harness as the primary workspace. An agent that can run `repro` and `test` is more valuable than an agent that can only read Slack.

Store diagnosis notes next to the harness: what was ruled out, not only what was true. Negative knowledge is part of the regression suite.

Tradeoffs

Full production replay is the most faithful and the most expensive to sanitize.

A minimized fixture is easier to keep, but you can minimize away the bug. Minimize after the failure is red, not before.

Recorded model outputs make AI-feature bugs deterministic; they also freeze a particular model version. Version the recordings.

Failure modes

Harnesses that only pass on one machine because of implicit timezones or filesystem layout.

Captures that contain secrets.

Tests that assert implementation details and break on every valid refactor.

Agent loops that patch the harness until it is green without fixing production.

Validation

The captured failure is red before the patch and green after — on CI.

A close cousin of the failure (off-by-one, duplicate event, delayed webhook) is also encoded, or explicitly listed as unhandled.

The harness does not require production credentials.

Lessons

AI agents do not remove systems engineering. They amplify whatever feedback loop you give them.

A reproducible failure is dramatically easier for humans and agents to diagnose. Make that the default, not a luxury.