Skip to main content

ADR 0045: Inspect Agent Traces Without Making Them Release Gates

Status​

Accepted.

Context​

STAGE 0.5.7 through 0.5.9 changed packaged skill routing, terminal stopping, and authority behavior. Static tests established that the intended instructions were present and structurally valid, but they could not establish how a fresh Codex consumer interpreted those instructions. Maintainers therefore ran fresh CLI probes and manually read JSONL traces. Those probes exposed real failures: unnecessary repository inspection, excessive token use, and an invented analogue after route rejection.

Manual trace reading was repetitive and easy to summarize imprecisely. Fully automating the probe, however, would conflate two different evidence classes. Agent behavior depends on the model, Codex version, installed skill snapshot, prompt, target state, and available tools. A sampled invocation is therefore nondeterministic and toolchain-bound. Literal output checks also cannot decide whether the response is semantically correct.

Decision​

Add a small Maintainer Laboratory command that inspects an already-recorded Codex --json trace.

The command:

  • does not invoke Codex or select a model;
  • requires a completed turn and completed agent message;
  • counts unique action items conservatively, including unknown item types;
  • can enforce optional action-call and input-token ceilings;
  • can require or forbid exact case-sensitive final-message literals;
  • fails closed when requested usage data is absent; and
  • always reports that semantic review is still required.

Use the command only when a packaged behavior change warrants a fresh consumer probe. Record the model, Codex version, prompt, installed plugin version, and target state alongside any conclusion. The inspector is not part of the deterministic release gate, and a passing trace is not a compatibility or semantic-correctness claim.

Do not retain raw traces in the repository by default. Keep them temporary unless a named release, research, or failure-analysis record needs their revision-bound evidence.

Consequences​

  • Maintainers can compare the structural cost and observable action boundary of behavior probes without hand-counting trace events.
  • Failed probes remain useful evidence rather than disappearing behind a final prose summary.
  • Repository tests can verify the inspector deterministically without invoking a model.
  • Semantic correctness, product authority, and interpretation remain human responsibilities.
  • Probe results cannot be generalized across models, Codex versions, plugin snapshots, prompts, targets, or tool availability without further evidence.
  • Releases are not blocked by unavailable or flaky model sampling unless a maintainer explicitly makes a named probe part of that release's evidence plan.
Last updated on