API workflow experiment
Eight real Codex trials worked through frozen responses from production Evals AX. A navigation index changed where the agent started; the task and source content stayed the same. The run exposed a defect in our verifier.
Known verifier defect: six false negatives
The original verifier recorded 0 of 8 successes. It rejected valid source quotations that expressed the evidence boundary without the literal word “inconclusive”. Six failed verdicts are known verifier false negatives; they must not be read as six agent failures.
| Assessment | Generic entry point | Navigation index |
|---|---|---|
| Original verifier v1 · known defect | 0 / 4 | 0 / 4 |
| Post-hoc verifier v2 · separate assessment | 2 / 4 | 4 / 4 |
The revised verifier accepts equivalent source-grounded quotations and preserves the other checks. All 8 retained answers were assessed; no model trials were rerun. Two answers still fail: one joins separate passages into a verbatim quote; the other supplies an unsupported CLI flag.
These counts do not establish that the navigation index improves performance. Four pairs are too few for a general claim, and the verifier changed after outcomes were observed.
Review every trial and its coverage
- 1-candidate: original failed; post-hoc passed — verifier false negative.
- 1-baseline: original failed; post-hoc failed — answer defect retained.
- 2-baseline: original failed; post-hoc passed — verifier false negative.
- 2-candidate: original failed; post-hoc passed — verifier false negative.
- 3-candidate: original failed; post-hoc passed — verifier false negative.
- 3-baseline: original failed; post-hoc passed — verifier false negative.
- 4-baseline: original failed; post-hoc failed — answer defect retained.
- 4-candidate: original failed; post-hoc passed — verifier false negative.
Real agent execution on frozen production response files, prepared for production-platform upload. No per-trial HTTP, authenticated account operations, browser or assistive-technology coverage.
Duration and usage are descriptive only; no efficiency inference was registered. Dollar cost is unknown in all eight trials.
Fresh local workspaces and retained provider traces are not external filesystem or complete-model-context attestation. Cloud storage does not add such attestation.
The task
Reconcile two public study records, calculate their descriptive measurements, and identify why those projections cannot be uploaded as full experiment results. Then prepare an authenticated account handoff: six API operations, the CLI commands, and source quotations supporting terms consent, result immutability and evidence limits.
The fixed protocol
Four randomized pairs; eight trials total. Each trial received a fresh copy of the source files and a 120-second budget. The candidate added only a navigation index; both conditions retained full documentation and identical production response bytes. Six independent verifier controls passed before execution. They missed valid equivalent quotations, which the later regression controls now cover.
Requested model: gpt-6-astra, low reasoning. Runtime: codex-cli 0.153.4. The run used the framework’s read-only Codex adapter with isolated configuration on an ordinary macOS host. All trials finished; no timeouts, runner errors, replacements or extra trials were excluded.
The plan was recorded at . Execution ran from to . Verifier v2 was specified after four trials had completed, then applied separately to all eight retained answers.
Observed time and usage
Duration and usage are descriptive only; no efficiency inference was registered. Dollar cost was unreported for all eight trials.
| Condition | Mean time | Input tokens | Output tokens | Dollar cost |
|---|---|---|---|---|
| Generic entry point | 106.91 s | 868,523 | 9,092 | Unknown (0 / 4 reported) |
| Navigation index | 97.97 s | 824,519 | 8,459 | Unknown (0 / 4 reported) |
What this covers
Real agent execution on frozen production response files, prepared for production-platform upload. No per-trial HTTP, authenticated account operations, browser or assistive-technology coverage.
The host captured public production responses, but a sandbox capability check could not resolve the production hostname. That failed check is retained; the model trials therefore processed frozen files. This does not test live network execution, an authenticated browser journey or assistive technology.
Fresh local workspaces and retained provider traces are not external filesystem or complete-model-context attestation. Cloud storage does not add such attestation.
This is a post-hoc verifier diagnosis and sensitivity analysis, not a preregistered success comparison. Neither the flawed original predicate nor the repaired counts establish that the navigation index improves agent performance.
The JSON is a review projection, not a replacement experiment result. Original verdicts remain unchanged.