Checking a complete dispatch
Reconciling a dispatch means resolving disagreements between files. This synthetic operations task makes those consequences verifiable; its first comparison also shows the limits of a small study.
In this article
A complete job with consequences you can check
The agent receives separate order and payment exports, stock movements, customer holds, and a dispatch policy. It must reconcile ten orders, subtract a refund, deduplicate exports, exclude conflicting stock entries, and allocate inventory in policy order. An unpaid order must not ship just because stock is available.
The required answer is a bundle containing ledger.csv, exceptions.json and dispatch.json. A frozen hand-reconciled oracle checks the records, exception coverage and remaining inventory. All data are synthetic; the task does not operate a warehouse, payment account or carrier service.
What happened in the first study
All 8 planned trials completed successfully through Codex codex-cli 0.153.4, requesting gpt-6-astra with low reasoning. No trials were retried, discarded or added after observing the outcomes.
| Recorded measure | Baseline | Indexed candidate |
|---|---|---|
| Complete task success | 4 / 4 | 4 / 4 |
| Mean agent duration | 52.26 s | 64.22 s |
| Reported input tokens | 243,090 | 266,419 |
| Reported output tokens | 4,317 | 5,845 |
| Dollar cost | Unknown | Unknown |
The success difference was zero, with the predeclared conservative 95% bound spanning −100 to +100 percentage points. This is inconclusive, not evidence that the two interfaces are equivalent. Timing and usage are descriptive; the larger candidate averages do not establish a penalty.
The first answer was also materialized into files: ten ledger orders, six exceptions and five shipments, with the independently expected remaining stock. This verifies the returned deliverable and the materializer; the read-only agent did not itself write those files.
Download the public evidence summary. The frozen source was committed at 5255a0cc9b8989fb07e34704d827baf955baca8b; the manifest hash is 53542810ac5f8f127fc3518b2e64f83dff0d6c94bd4c8a70be1309af9e194823. Original evidence remains in the framework repository alongside the recorded plan.
Keep the limits attached
Ordinary local execution was not externally observed. Complete model input and access outside the trial workspace remain unobserved. Evidence is unsigned and client-reported.
The frozen study used verifier version 1 and four controls. The current example's verifier version 2 accepts equivalent integer spellings and adds refund and partial-shipment negative controls. Those later repairs do not alter this study's results or coverage.
The original known-answer control and negative controls for an unpaid shipment, omitted exception and double-counted stock matched the verifier. Those controls qualify particular cases; they do not prove an infallible acceptance predicate. The later equivalent-serialization regression is a concrete example of improving the framework when its verifier rejects something it should accept.
Four pairs have little power under this conservative bound. Use a representative task collection, an independently reviewed verifier and a declared sampling plan before making a release-wide claim. This task demonstrates how to perform a complete experiment; it does not establish that indexes help agents in general.