Choose a task worth measuring

Reconcile an operations handoff

An operations agent has to join orders with inventory, interpret dispatch policy, and produce a ledger with an exception list. A plausible summary is insufficient: a wrong allocation can leave an order unfulfilled or hide a discrepancy.

Compare the same underlying data in a baseline folder and a candidate with a consistent data dictionary and navigable index. Keep every source file accessible in both. Verify the output against record identifiers, arithmetic, stock constraints, and the stated policy. A verifier should reject a dropped exception even when the totals happen to balance.

Use a frozen export and a proposed dispatch plan for the experiment. Execution of shipments or other external actions is a separate authorization boundary. The framework's local read-only adapters can evaluate the returned deliverable without granting production-system write access.

Our runnable dispatch experiment includes synthetic exports and an independent acceptance oracle. Eight actual Codex trials completed the handoff; both conditions passed equally, so the comparison is inconclusive.

Test the context around a codebase

A maintainer changes AGENTS.md, splits a large skill, or adds an index to a repository. Measure whether an agent can find the relevant implementation and finish representative tasks. Keep source content available, and use acceptance tests or exact source references to check its answer.

Do not assume a shorter file improves autonomy. Our first context-discovery study found no success difference in its small recorded comparison. The result was inconclusive, and intermediate failed reads remained useful diagnostic observations. Current external studies also show context effects depend on the evaluated tasks and agents; see the research review.

Change a tool contract without breaking the job

An agent must find a customer, retrieve paginated records, identify an inconsistent entry, and prepare a correction. Compare a revised tool schema or error response with the current version using the same seeded data. Verify the intended record and correction, including the no-change case.

A replay or test tenant lets you check error recovery, pagination, and permission boundaries without changing live customer records. Integrate your trusted harness with the experiment contract. This is an experiment design you can implement; early access does not provision a hosted copy of your application or promise universal MCP compatibility.

Evaluate a complete browser workflow

Choose a job such as filtering a date range, exporting records, and checking that the export matches the selected account. Change one relevant interface property and repeat the task with the actual browser agent. Check the exported artifact and final state rather than the agent's claim that it clicked Export.

A Vibecheck can suggest an initial hypothesis about the public HTML. It cannot test authentication, JavaScript state, frames, or the completed workflow. Supply those measurements through your own trusted runner and independent verifier. Keep model configuration and browser state fixed, and report incomplete trials.

Make the result useful to a release decision

Record what would justify shipping the change before you run it. Compare verified completion with elapsed time and measured usage. Lower token use on a failed task is not a saving. If the result is uncertain, preserve it and identify the additional tasks or repetitions that would resolve the decision.

The CLI runs local experiments; the platform stores their manifests and results. Web and CLI account operations use the same API, so evidence stays available when the agent session ends.