# Checking a complete dispatch

Reconciling a dispatch means resolving disagreements between files. This synthetic operations task makes those consequences verifiable; its first comparison also shows the limits of a small study.

## A complete job with consequences you can check

The agent receives separate order and payment exports, stock movements, customer holds, and a dispatch policy. It must reconcile ten orders, subtract a refund, deduplicate exports, exclude conflicting stock entries, and allocate inventory in policy order. An unpaid order must not ship just because stock is available.

The required answer is a bundle containing ledger.csv, exceptions.json and dispatch.json. A frozen hand-reconciled oracle checks the records, exception coverage and remaining inventory. All data are synthetic; the task does not operate a warehouse, payment account or carrier service.

## What happened in the first study

All 8 planned trials completed successfully through Codex codex-cli 0.153.4, requesting gpt-6-astra with low reasoning. No trials were retried, discarded or added after observing the outcomes.

| Recorded measure | Baseline | Indexed candidate |
| --- | --- | --- |
| Complete task success | 4 / 4 | 4 / 4 |
| Mean agent duration | 52.26 s | 64.22 s |
| Reported input tokens | 243,090 | 266,419 |
| Reported output tokens | 4,317 | 5,845 |
| Dollar cost | Unknown | Unknown |

The success difference was zero, with the predeclared conservative 95% bound spanning −100 to +100 percentage points. This is inconclusive, not evidence that the two interfaces are equivalent. Timing and usage are descriptive; the larger candidate averages do not establish a penalty.

The first answer was also materialized into files: ten ledger orders, six exceptions and five shipments, with the independently expected remaining stock. This verifies the returned deliverable and the materializer; the read-only agent did not itself write those files.

[Download the public evidence summary](/evidence/dispatch-2026-09-08.json). The frozen source was committed at 5255a0cc9b8989fb07e34704d827baf955baca8b; the manifest hash is 53542810ac5f8f127fc3518b2e64f83dff0d6c94bd4c8a70be1309af9e194823. Original evidence remains in the framework repository alongside the recorded plan.

## Keep the limits attached

Ordinary local execution was not externally observed. Complete model input and access outside the trial workspace remain unobserved. Evidence is unsigned and client-reported.

The frozen study used verifier version 1 and four controls. The current example's verifier version 2 accepts equivalent integer spellings and adds refund and partial-shipment negative controls. Those later repairs do not alter this study's results or coverage.

The original known-answer control and negative controls for an unpaid shipment, omitted exception and double-counted stock matched the verifier. Those controls qualify particular cases; they do not prove an infallible acceptance predicate. The later equivalent-serialization regression is a concrete example of improving the framework when its verifier rejects something it should accept.

Four pairs have little power under this conservative bound. Use a representative task collection, an independently reviewed verifier and a declared sampling plan before making a release-wide claim. This task demonstrates how to perform a complete experiment; it does not establish that indexes help agents in general.

## About this article

Evals AX · Experiment

Study conducted: 8 September 2026

- [Public evidence JSON](/evidence/dispatch-2026-09-08.json)
- [Fixed sampling plan](https://github.com/evals-ax/website/blob/8a16db785ade840b7cc8320e2c316b8beb7ad826/packages/framework/evidence/dispatch-2026-09-08-plan.md)
- [Run the dispatch example](/docs/dispatch#run)
- [Complete source documentation](/docs/dispatch)
- [Full documentation as Markdown](/docs/full.md)