Define success
Choose a real task. Write a verifier that checks its outcome independently of the agent.
THE EVALS AX FRAMEWORK / EARLY ACCESS
The agent stays.
The world changes.
The same agent can succeed or struggle depending on the environment around it. Find out what actually makes a difference.
Establish a baseline
Record the agent configuration and environment. Decide how many trials to run before seeing results.
Change the interface
Test a specific hypothesis against the baseline. Keep unrelated conditions comparable.
Repeat and compare
Retain failures as evidence. Report the measured effect and its uncertainty, including an inconclusive result.
Test the environment around the agent.
Run a baseline and a candidate with a fixed task and agent configuration. The local framework supports Codex, Claude Code, or an explicit custom command. It creates fresh workspaces for paired trials and runs a separate verifier to check the outcome. You bring the runtime and control its execution budget.
The result preserves the experiment conditions and each trial’s evidence. Comparisons include uncertainty; a small experiment may leave the question unresolved. Evidence about one task does not establish a rule for every agent or environment.
Context is part of the environment.
The instructions an agent reads and the information it can discover are testable interface choices. The context scanner surfaces concrete issues in local files, with paths and line evidence. It can suggest a question; a controlled experiment tests whether the change improves the outcome.
Thin skills, progressive disclosure, and access to full context are promising ideas to investigate. Their value depends on the task. We apply the same standard to our own framework and docs.
Our first study tested a source index with Codex. Every trial succeeded, leaving the success comparison inconclusive. The traces still revealed recovery work the final score missed. Read the study and its limits →
A later experiment asked Codex to assemble an API workflow from our production docs. It exposed six false negatives in the verifier. Inspect the eight trials and the separate correction →
Start with a website observation.
Vibecheck inspects a page’s server-delivered HTML and public companion files. Its 36 checks cover semantics, noise, structured data, interactive elements, discoverability, rendering dependency and protocol hygiene. Each report records the auditor version and the evidence behind its findings.
It does not run a browser, execute JavaScript, log into a site, or measure an agent completing a task. Its observations are a starting point for investigation, not a measured probability of agent success. Different agents and tasks may benefit from different changes.
From a finding to an experiment
Suppose Vibecheck finds an unlabeled checkout button. Add a clear accessible name, rerun the static audit, then test the same checkout task with a fixed agent configuration across both interface versions. The first step verifies the markup changed. The second measures whether the change helped.
A method that can travel.
The same experimental logic applies to a tool response or a physical workspace. Early access provides a local command runner; a physical experiment still needs its own instrumentation and repeatable reset. We do not infer physical-world performance from a website score.