# Run a local experiment

Bring your agent runtime. Preserve the task, environment, and outcome needed to interpret each trial.

## The experiment contract

An experiment manifest records a hypothesis, a task, baseline and candidate variants, the agent configuration, an independent verifier, and a fixed repetition count. The framework validates the manifest before execution. Keep the manifest in version control beside the environment it describes.

```sh
evx experiment init ./my-experiment
evx experiment validate ./my-experiment/manifest.json
evx experiment run ./my-experiment/manifest.json --output ./results
```

Built-in Codex and Claude Code adapters are read-only in this release: Codex uses its read-only sandbox and Claude is limited to reading and searching files. An explicit custom command can integrate other trusted local runtimes. A fresh copied workspace provides repeatability; it does not isolate an arbitrary custom command from the host.

Local execution requires macOS or Linux, including WSL. Native Windows execution stops before starting subprocesses or creating trial artifacts because this release cannot enforce Windows process-tree cleanup. Analysis, context inspection and account commands do not require local experiment execution.

## Verify the outcome independently

The verifier runs outside the agent's workspace and returns a structured result. It should inspect the actual task outcome: for example, execute acceptance tests or check an artifact against a specification. Do not substitute the agent's confidence or final message for verification.

```json
{ "passed": true, "reason": "The output satisfies the task specification", "metrics": {} }
```

Independent process execution prevents accidental coupling; it does not make an arbitrary verifier correct or tamper-proof. Review what the verifier can observe, and test it against intentionally wrong outputs.

The manifest supports positive and negative verifier controls. When configured, they run before the first agent trial; a verifier that accepts the wrong-answer control fails qualification.

## Read the comparison

Our [API workflow experiment](/evidence/production-api-handoff-2026-09-08) shows eight real Codex trials using captured production responses. It also shows why verifier coverage matters: the original verifier rejected valid quotations. The recorded verdicts remain unchanged, with a separate assessment of every answer and no claimed performance improvement.

The runner randomizes baseline/candidate order within each pair from a recorded seed. Every trial starts from a fresh copy. Agent errors and timeouts count as failures. Skipped or cancelled work remains visible and prevents a complete comparison claim. Verifier errors also suppress inference because the outcome was not reliably measured.

The reported effect is the candidate success fraction minus the baseline success fraction. The conservative 95% Hoeffding interval assumes independent pairs. The exact two-sided discordant-pair test is descriptive; it does not correct for trying many hypotheses or establish generalization.

Repetitions range from 2 to 100 per variant in early access. Small runs are useful for diagnosing the setup, but will often be inconclusive. Choose a sample size based on the effect that matters and the available budget, not on whether an early result looks favorable.

## Save automatically

```sh
evx auth status
evx projects list
evx experiment run ./my-experiment/manifest.json --output ./results --project PROJECT_ID
```

With evx 0.7.1 or later, a connected host saves the declaration and result directly to the selected project. Run `evx auth login` once if the host is not connected. Create a project with `evx projects create --name 'Interface experiments'`, or use the project ID shown on its web page. The website is for inspecting the evidence; no file upload or pasted JSON is required.

The manifest still belongs in your local project beside its surface and verifier files. Running it executes those declared commands with your agent/provider account and may incur costs. Results can include prompts, local paths and verifier messages. To keep the experiment on the host, replace `--project PROJECT_ID` or `--experiment EXPERIMENT_ID` with `--local-only`.

```sh
evx experiment run ./my-experiment/manifest.json --output ./results --experiment EXPERIMENT_ID
```

Use `--experiment` for an existing declaration, with its matching local manifest and workspace. A downloaded manifest does not include the files needed to execute it. If a save fails after execution, keep the output directory and use the exact retry command returned by evx; do not repeat the trials to retry a save.

Saved results are immutable. A changed intervention needs a new declaration. The service validates the submitted trial contract and recomputes its comparison; it does not attest that external execution occurred or certify the verifier's coverage. Raw trace files are currently retained separately on the host.