Every task passed. There was still more to learn.

Would a direct source index help?

We asked Codex to recover exact facts about Evals AX authentication from a selected snapshot of our shipped code and architecture documentation. The candidate added an instruction-file index pointing to canonical implementation files. Both conditions contained the same underlying source and documentation.

The local plan was recorded at 2026-09-07T08:14:18.471Z, before execution began at 2026-09-07T08:14:18.474Z. Verified task success was the primary outcome. Duration and provider-reported usage were declared secondary descriptive measurements. This is a locally recorded plan, not independent third-party registration.

The success comparison was inconclusive

The observed candidate-minus-baseline success difference was 0 percentage points. The conservative 95% paired Hoeffding interval ranged from -78.4 to +78.4 percentage points, assuming independent pairs.

Every trial succeeded, leaving this task at a ceiling for the chosen success criterion. The result does not establish equivalence, demonstrate an improvement, or show that an index is unnecessary. A harder or more varied task set is needed to examine success differences.

The confidence calculation assumes independent repeated pairs. Fresh workspaces did not reset provider caches or eliminate service drift; those are limits on interpreting the interval.

Other measurements are descriptive

No inferential comparison of efficiency was planned. The timing and token totals do not establish that a source index makes agents slower or more expensive. Dollar cost is unknown, not zero; token counts are not a provider invoice.

In the candidate condition, each trial attempted an API route path absent from the selected snapshot and recovered through search. The index linked to files present in the snapshot; the agent inferred the missing route paths from the source structure.

A separate post hoc extraction counted 24 completed commands and 0 nonzero exits in the baseline, compared with 36 commands and 19 nonzero exits in the candidate. Those candidate exits included missing-file reads and searches with no matches; a nonzero exit is not automatically an error.

That snapshot limitation matters. A failure to read an absent route in a selected snapshot is not evidence about the same task in the full repository, and the added index itself had no broken links.

A passing task can hide recovery work

The final success metric missed the intermediate failed reads because the agent recovered and returned correct answers. We added provider-event diagnostics that retain command counts and nonzero exit positions without copying raw command text into uploaded results. Adapters without diagnostic coverage report unknown values.

The original result remains unchanged. The later diagnostic extraction is recorded separately, rather than presented as a measurement captured by the original runner. A failed tool command is still an observation; calling it confusion or intent misalignment requires further evidence.

Inspect the public record

The aggregate study record contains the configuration, source revision identifiers, manifest hash, and numerical comparison. It is derived from the framework's retained result and summary. Source snapshots and raw traces are not included in that public response.

Manifest hash: cc6afc0f7888bfcbd201fa93b70f18c11289f55d34591a7add6bce2dfe646fef.

This was a selected authentication-source snapshot, not the full repository. The observations do not establish that indexes help or harm agents in other environments. Local plan timestamps and client-produced evidence are not independent third-party attestation.