# Reconcile a dispatch, end to end

A runnable operations task with a frozen oracle, real agent trials, and an inconclusive comparison.

## A complete job with consequences you can check

The agent receives separate order and payment exports, stock movements, customer holds, and a dispatch policy. It must reconcile ten orders, subtract a refund, deduplicate exports, exclude conflicting stock entries, and allocate inventory in policy order. An unpaid order must not ship just because stock is available.

The required answer is a bundle containing ledger.csv, exceptions.json and dispatch.json. A frozen hand-reconciled oracle checks the records, exception coverage and remaining inventory. All data are synthetic; the task does not operate a warehouse, payment account or carrier service.

## Run it on your host

[Install the CLI](/docs/cli#install). The example needs Node 22.18 or later and an authenticated Codex host with the requested model available. Inspect the manifest before running it: four randomized pairs allow 100 seconds per trial and 15 minutes overall. Your provider/account allowance pays for agent execution.

```sh
evx experiment init ./dispatch-study --template dispatch-reconciliation
cd dispatch-study
evx experiment validate manifest.json
evx experiment run manifest.json --output ./results
evx experiment validate ./results/result.json
evx experiment compare ./results/result.json
```

The baseline has a generic entrypoint. The candidate adds a source/join-key index; every other source file remains identical and accessible. Changing the task, runtime or budget creates a different experiment configuration. Keep that change with the resulting evidence.

```sh
evx experiment output ./results/result.json --trial 1-baseline > answer.json
node materialize.mjs answer.json ./dispatch-output
```

The explicit output command verifies the selected retained trace and answer hashes before printing the agent's answer. It does not verify every trial or an external observer receipt. The separate materializer only writes an independently passing answer, creates a fresh directory, and refuses to overwrite an existing one. Inspect local answers before sharing them; this command deliberately reveals their content.

## What happened in the first study

All 8 planned trials completed successfully through Codex codex-cli 0.153.4, requesting gpt-6-astra with low reasoning. No trials were retried, discarded or added after observing the outcomes.

| Recorded measure | Baseline | Indexed candidate |
| --- | --- | --- |
| Complete task success | 4 / 4 | 4 / 4 |
| Mean agent duration | 52.26 s | 64.22 s |
| Reported input tokens | 243,090 | 266,419 |
| Reported output tokens | 4,317 | 5,845 |
| Dollar cost | Unknown | Unknown |

The success difference was zero, with the predeclared conservative 95% bound spanning −100 to +100 percentage points. This is inconclusive, not evidence that the two interfaces are equivalent. Timing and usage are descriptive; the larger candidate averages do not establish a penalty.

The first answer was also materialized into files: ten ledger orders, six exceptions and five shipments, with the independently expected remaining stock. This verifies the returned deliverable and the materializer; the read-only agent did not itself write those files.

[Download the public evidence summary](/evidence/dispatch-2026-09-08.json). The frozen source was committed at 5255a0cc9b8989fb07e34704d827baf955baca8b; the manifest hash is 53542810ac5f8f127fc3518b2e64f83dff0d6c94bd4c8a70be1309af9e194823. Original evidence remains in the framework repository alongside the recorded plan.

## Keep the limits attached

Ordinary local execution was not externally observed. Complete model input and access outside the trial workspace remain unobserved. Evidence is unsigned and client-reported.

The frozen study used verifier version 1 and four controls. The current example's verifier version 2 accepts equivalent integer spellings and adds refund and partial-shipment negative controls. Those later repairs do not alter this study's results or coverage.

The original known-answer control and negative controls for an unpaid shipment, omitted exception and double-counted stock matched the verifier. Those controls qualify particular cases; they do not prove an infallible acceptance predicate. The later equivalent-serialization regression is a concrete example of improving the framework when its verifier rejects something it should accept.

Four pairs have little power under this conservative bound. Use a representative task collection, an independently reviewed verifier and a declared sampling plan before making a release-wide claim. This task demonstrates how to perform a complete experiment; it does not establish that indexes help agents in general.