# Start with a question Find a concrete obstacle, then test whether removing it helps an agent finish its task. ## Who this is for Evals AX is for people building the environments agents work in: tool authors, teams maintaining codebases, and product builders opening their interfaces to agents. A coding agent can use the CLI directly during development. The framework runs experiments on your machine with your agent runtime. The platform keeps projects and experiment records alongside your website audits. Early access does not execute models in the cloud. ## Inspect a website Sign in, accept the current [terms](/account/terms), then enter a public URL in [Vibecheck](/vibecheck). For CLI access, [authenticate the host](/docs/cli#authentication) first. Reports contain observations from public HTML and companion files. Local inspection and public report reads do not require an account. ### Bun (default) ```sh bunx --package @evals-ax/evx@https://evals.ax/downloads/evx-0.7.1.tgz evx vibecheck https://example.com --wait ``` ### npm ```sh npx --yes https://evals.ax/downloads/evx-0.7.1.tgz vibecheck https://example.com --wait ``` ### pnpm ```sh pnpm dlx https://evals.ax/downloads/evx-0.7.1.tgz vibecheck https://example.com --wait ``` ### Yarn 4 ```sh yarn dlx -p @evals-ax/evx@https://evals.ax/downloads/evx-0.7.1.tgz evx vibecheck https://example.com --wait ``` The current auditor has 36 checks. Reports distinguish partial standards checks from unvalidated hypotheses and optional conventions. A report without flags is not proof that an interface is effective. Legacy scores remain in JSON for compatibility; they do not predict agent success. ## Test a proposed change Choose a task with a result you can independently verify. Prepare a baseline environment and a candidate that changes the interface. Record the hypothesis before the run. The [experiment guide](/docs/experiments) explains the manifest, local execution, and result format. ```sh evx experiment init ./my-experiment evx experiment validate ./my-experiment/manifest.json evx experiment run ./my-experiment/manifest.json --output ./results ``` The starter is a fixture for learning the contract. Replace its task and verifier with your own before treating the result as evidence about a real agent. ## Keep your work [Create a free account](/user/sign-up) to keep report history and private project records. Use [CLI sign-in](/docs/cli#authentication) to connect a host machine. Public website report links can be shared; project and experiment records require account access. Already have reports saved in this browser? The platform asks explicitly before attaching them to the signed-in account. --- # Website histories Keep every retained run for a URL together and compare matching observations. ## One property per URL The [Websites](/dashboard) page groups your runs by the normalized requested URL. Opening a property shows its latest result, the most recent comparable pair and a paginated history. Grouping covers all retained runs, not just the first page. HTTP and HTTPS remain separate, as do different paths and query strings. Fragments are removed by the existing URL normalizer. Property keys identify URLs; they do not grant access to another account’s history. ## Run and compare Rerun & compare immediately starts an audit of the same URL with the displayed report as its baseline. The new report opens while the audit runs. Merely opening a property or report never starts an audit. ```sh evx vibecheck https://example.com --compare-to REPORT_ID --wait ``` Comparisons require matching targets, auditors and methodologies. A version change can make runs incomparable. Coverage changes are separate from measured check changes; neither establishes an effect on agent performance. Legacy reports remain accessible in their original form. ## Read history through the API ```sh evx api GET /websites evx api GET /websites/PROPERTY_KEY ``` Use the returned nextCursor unchanged to fetch the next page. Website collections and histories require account access through the API. The web interface also supports browser-owned guest history. Individual website report links remain shareable; the grouped history is private. Delete removes an individual report’s content and its public link. Deleting the final retained run removes that property from the overview. Creating more reports does not overwrite earlier runs. --- # API, CLI and MCP inspections Collect bounded interface observations on your host, inspect the evidence, and save private reports. ## Start with an explicit plan Write a JSON plan, then run evx inspect. With a CLI login, the completed report saves privately to your account automatically; without one, it stays local. Add --local-only to opt out for any run. Reports can include interface names and diagnostic evidence. The CLI executes only the probes you select. An inspection is a set of interface observations, not an agent benchmark or a universal readiness score. ```sh evx inspect ./inspection.json --output ./report.json ``` The plan bounds repetitions, per-operation time, total time and output size. Review commands before running them: starting a local program can have side effects even when a probe is named help. No local commands are executed by the hosted service. ## Inspect an API The API collector reads an OpenAPI JSON document and inventories its operations and schema descriptions. Optional GET or HEAD probes must be selected explicitly and remain on the configured origin. This example checks that an anonymous account request is rejected; it does not create an account or an audit. ```json { "schemaVersion": "1", "kind": "api", "name": "Evals AX public API", "target": { "url": "https://evals.ax/api/v1/openapi.json" }, "probes": [ { "path": "/api/v1/account", "method": "GET", "expectedStatuses": [ 401 ] } ] } ``` A successful HTTP response does not establish that an agent can complete an API workflow. Check response meaning with an independent verifier in an experiment when that is the question. Authentication headers can reference environment variable names through headersFromEnv; credential values do not belong in the plan. ## Inspect a CLI Choose exact argument arrays and expected exit codes. Processes run without a shell. The collector records bounded structural observations and output hashes; it does not upload raw process output by default. This example requires evx to be installed on PATH. ```json { "schemaVersion": "1", "kind": "cli", "name": "evx command contract", "probes": [ { "id": "help", "purpose": "help", "command": [ "evx", "--help" ], "expectedExitCodes": [ 0 ] }, { "id": "version", "purpose": "version", "command": [ "evx", "--version" ], "expectedExitCodes": [ 0 ] }, { "id": "invalid-argument", "purpose": "invalid-input", "command": [ "evx", "--not-a-real-option" ], "expectedExitCodes": [ 1 ] } ] } ``` A help flag and a nonzero invalid-input exit code are observations. They do not prove instructions are understandable or that the command is safe or useful. Use custom probes for the contract your tool actually promises. CLI probes and MCP stdio transports can name host variables in env; their values are passed to the process and redacted from report text as credentials. Use publicEnv only for nonsecret settings such as NO_COLOR. Public values are not redacted, so they must never contain credentials. The plan stores variable names, not their values. ## Inspect an MCP server MCP inspections discover the server and list tool definitions without calling its tools. Set the exact protocol revision your server supports. The collector supports 2026-07-28 discovery and the older 2025-11-25 initialization flow; it does not silently downgrade. Replace the command below with a server you intend to start. ```json { "schemaVersion": "1", "kind": "mcp", "name": "My MCP server", "protocolVersion": "2026-07-28", "transport": { "type": "stdio", "command": [ "node", "./server.mjs" ] }, "maxPages": 5 } ``` HTTP transport uses type http with url and optional headersFromEnv in place of command. Pagination is bounded by maxPages. Missing descriptions and schema observations are leads for investigation, not demonstrated causes of agent failure. See the [MCP versioning specification](https://modelcontextprotocol.io/specification/2026-07-28/basic/versioning). ## Save and inspect the report ```sh evx auth login evx inspect ./inspection.json --output ./report.json --title "Production API discovery" evx interfaces list ``` After CLI sign-in, inspections appear in [API, CLI & MCP reports](/dashboard/interfaces) as part of the run. The CLI writes and flushes the local file first, then saves through the shared API. Its JSON response includes cloud.status and cloud.reportUrl; saved confirms the account accepted the report, while local-only identifies an explicit opt-out or missing CLI login. Reports remain private and immutable. The service validates the contract and derives the displayed counts; it does not attest that host execution occurred. For sensitive interfaces, use --local-only to review the report before sharing it. Retained reports count toward the account storage allowance and can be deleted. Raw local traces remain on the host. Experiment uploads and provider telemetry still require their separate explicit commands. A save failure exits with code 1 and INSPECTION_SAVE_FAILED on stderr, keeping the local report. The error includes its path, report ID, safe reason and a retry command with an argument array. A network failure leaves cloud delivery unconfirmed; retrying the same retained report and title is idempotent. No inspection is rerun and no write is retried automatically. A 428 response links to the current terms for actual account acceptance; the CLI cannot accept them for you. Cancellation retains local evidence and exits 130. --- # Choose a task worth measuring Test the environment around an agent against a complete job and a result you can check. ## Reconcile an operations handoff An operations agent has to join orders with inventory, interpret dispatch policy, and produce a ledger with an exception list. A plausible summary is insufficient: a wrong allocation can leave an order unfulfilled or hide a discrepancy. Compare the same underlying data in a baseline folder and a candidate with a consistent data dictionary and navigable index. Keep every source file accessible in both. Verify the output against record identifiers, arithmetic, stock constraints, and the stated policy. A verifier should reject a dropped exception even when the totals happen to balance. Use a frozen export and a proposed dispatch plan for the experiment. Execution of shipments or other external actions is a separate authorization boundary. The framework's local read-only adapters can evaluate the returned deliverable without granting production-system write access. Our [runnable dispatch experiment](/docs/dispatch) includes synthetic exports and an independent acceptance oracle. Eight actual Codex trials completed the handoff; both conditions passed equally, so the comparison is inconclusive. ## Test the context around a codebase A maintainer changes AGENTS.md, splits a large skill, or adds an index to a repository. Measure whether an agent can find the relevant implementation and finish representative tasks. Keep source content available, and use acceptance tests or exact source references to check its answer. Do not assume a shorter file improves autonomy. Our [first context-discovery study](/docs/evidence) found no success difference in its small recorded comparison. The result was inconclusive, and intermediate failed reads remained useful diagnostic observations. Current external studies also show context effects depend on the evaluated tasks and agents; see [the research review](/docs/vibecheck-research). ## Change a tool contract without breaking the job An agent must find a customer, retrieve paginated records, identify an inconsistent entry, and prepare a correction. Compare a revised tool schema or error response with the current version using the same seeded data. Verify the intended record and correction, including the no-change case. A replay or test tenant lets you check error recovery, pagination, and permission boundaries without changing live customer records. Integrate your trusted harness with the [experiment contract](/docs/experiments). This is an experiment design you can implement; early access does not provision a hosted copy of your application or promise universal MCP compatibility. ## Evaluate a complete browser workflow Choose a job such as filtering a date range, exporting records, and checking that the export matches the selected account. Change one relevant interface property and repeat the task with the actual browser agent. Check the exported artifact and final state rather than the agent's claim that it clicked Export. A [Vibecheck](/vibecheck) can suggest an initial hypothesis about the public HTML. It cannot test authentication, JavaScript state, frames, or the completed workflow. Supply those measurements through your own trusted runner and independent verifier. Keep model configuration and browser state fixed, and report incomplete trials. ## Make the result useful to a release decision Record what would justify shipping the change before you run it. Compare verified completion with elapsed time and measured usage. Lower token use on a failed task is not a saving. If the result is uncertain, preserve it and identify the additional tasks or repetitions that would resolve the decision. The [CLI](/docs/cli) runs local experiments; the [platform](/dashboard/projects) stores their manifests and results. Web and CLI account operations use the same [API](/docs/api), so evidence stays available when the agent session ends. --- # Reconcile a dispatch, end to end A runnable operations task with a frozen oracle, real agent trials, and an inconclusive comparison. ## A complete job with consequences you can check The agent receives separate order and payment exports, stock movements, customer holds, and a dispatch policy. It must reconcile ten orders, subtract a refund, deduplicate exports, exclude conflicting stock entries, and allocate inventory in policy order. An unpaid order must not ship just because stock is available. The required answer is a bundle containing ledger.csv, exceptions.json and dispatch.json. A frozen hand-reconciled oracle checks the records, exception coverage and remaining inventory. All data are synthetic; the task does not operate a warehouse, payment account or carrier service. ## Run it on your host [Install the CLI](/docs/cli#install). The example needs Node 22.18 or later and an authenticated Codex host with the requested model available. Inspect the manifest before running it: four randomized pairs allow 100 seconds per trial and 15 minutes overall. Your provider/account allowance pays for agent execution. ```sh evx experiment init ./dispatch-study --template dispatch-reconciliation cd dispatch-study evx experiment validate manifest.json evx experiment run manifest.json --output ./results evx experiment validate ./results/result.json evx experiment compare ./results/result.json ``` The baseline has a generic entrypoint. The candidate adds a source/join-key index; every other source file remains identical and accessible. Changing the task, runtime or budget creates a different experiment configuration. Keep that change with the resulting evidence. ```sh evx experiment output ./results/result.json --trial 1-baseline > answer.json node materialize.mjs answer.json ./dispatch-output ``` The explicit output command verifies the selected retained trace and answer hashes before printing the agent's answer. It does not verify every trial or an external observer receipt. The separate materializer only writes an independently passing answer, creates a fresh directory, and refuses to overwrite an existing one. Inspect local answers before sharing them; this command deliberately reveals their content. ## What happened in the first study All 8 planned trials completed successfully through Codex codex-cli 0.153.4, requesting gpt-6-astra with low reasoning. No trials were retried, discarded or added after observing the outcomes. | Recorded measure | Baseline | Indexed candidate | | --- | --- | --- | | Complete task success | 4 / 4 | 4 / 4 | | Mean agent duration | 52.26 s | 64.22 s | | Reported input tokens | 243,090 | 266,419 | | Reported output tokens | 4,317 | 5,845 | | Dollar cost | Unknown | Unknown | The success difference was zero, with the predeclared conservative 95% bound spanning −100 to +100 percentage points. This is inconclusive, not evidence that the two interfaces are equivalent. Timing and usage are descriptive; the larger candidate averages do not establish a penalty. The first answer was also materialized into files: ten ledger orders, six exceptions and five shipments, with the independently expected remaining stock. This verifies the returned deliverable and the materializer; the read-only agent did not itself write those files. [Download the public evidence summary](/evidence/dispatch-2026-09-08.json). The frozen source was committed at 5255a0cc9b8989fb07e34704d827baf955baca8b; the manifest hash is 53542810ac5f8f127fc3518b2e64f83dff0d6c94bd4c8a70be1309af9e194823. Original evidence remains in the framework repository alongside the recorded plan. ## Keep the limits attached Ordinary local execution was not externally observed. Complete model input and access outside the trial workspace remain unobserved. Evidence is unsigned and client-reported. The frozen study used verifier version 1 and four controls. The current example's verifier version 2 accepts equivalent integer spellings and adds refund and partial-shipment negative controls. Those later repairs do not alter this study's results or coverage. The original known-answer control and negative controls for an unpaid shipment, omitted exception and double-counted stock matched the verifier. Those controls qualify particular cases; they do not prove an infallible acceptance predicate. The later equivalent-serialization regression is a concrete example of improving the framework when its verifier rejects something it should accept. Four pairs have little power under this conservative bound. Use a representative task collection, an independently reviewed verifier and a declared sampling plan before making a release-wide claim. This task demonstrates how to perform a complete experiment; it does not establish that indexes help agents in general. --- # Vibecheck's evidence boundary Research review: September 8, 2026 (Europe/London). Static findings suggest experiments; they do not predict agent success. ## Measure the interface the agent actually receives Modern agents can receive a browser accessibility tree, screenshots, transformed page content, or structured tool responses. Raw HTML length is a different measurement. Vibecheck inspects a fetched document and origin companion files; it does not observe those other representations or complete a task. The [BrowserGym implementation](https://github.com/ServiceNow/BrowserGym) provides task-based browser evaluation infrastructure. It supports the need for explicit observation and outcome contracts; it does not validate Vibecheck's historical HTML ratios or numerical cutoffs. ## Optional conventions need a named consumer Google's [current generative search guidance](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) says llms.txt and special structured data are not prerequisites for its generative search features. That is evidence about Google Search, not every agent. Before adding such a file to improve an agent workflow, establish whether the intended client requests and uses it. Chrome's [WebMCP announcement](https://developer.chrome.com/blog/webmcp-epp) describes an early preview for explicit browser tools. This is a promising experiment surface, not evidence that every agent supports it or that the advertised tools are correct and safe. ## Context design remains an experimental question The June 2026 revision of [Evaluating AGENTS.md](https://arxiv.org/abs/2602.11988v2) challenges the assumption that adding repository context generally improves coding outcomes. A July [two-agent ablation study](https://arxiv.org/abs/2607.27250v1) also found no measurable correctness benefit within its reported task set and bounds. These preprints do not establish that useful instructions should always be removed. Anthropic's [advanced tool-use engineering report](https://www.anthropic.com/engineering/advanced-tool-use) describes benefits and tradeoffs from deferred discovery and programmatic orchestration on its workloads. Discovery adds work too. Test your actual tool library, preserve access to full context, and check for instructions that contradict each other. ## Read findings with their limits Accessibility standards provide a basis for inspecting names and semantics, but our bounded static checks do not certify WCAG conformance or reproduce a browser's complete accessible-name algorithm. The current [Accessible Name 1.2 text](https://www.w3.org/TR/2026/WD-accname-1.2-20260827/) is a working draft. Historical scores retain uncalibrated checklist weights and thresholds. A higher score is not evidence that an agent became more effective. Keep the report version with the observation, inspect missing coverage, and use [independently verified comparisons](/docs/experiments) for claims about task outcomes. --- # What counts as evidence Keep the observation separate from the explanation. Make an improvement claim earn its place. ## A finding is a question worth testing An observation describes something the system measured: a control has no accessible name, an instruction file links to a missing path, or a verifier rejected an agent's output. A hypothesis proposes an explanation. An experiment estimates the effect of a specified change under recorded conditions. These are different evidence types. Static audits can find defects and suggest experiments. They do not establish a causal relationship between a design pattern and agent success. ## Define the comparison before running it 1. Write the task and the observable success criterion. Use a verifier independent of the agent's own claim of completion. 2. Record the agent configuration and environment versions. Set the repetitions and budgets before inspecting outcomes. 3. Change one proposed cause where practical. Keep unrelated task conditions comparable and reset the environment for each trial. 4. Keep failures and incomplete runs in the record. Compare the planned sample, report uncertainty, and inspect what happened. Changing several things at once can test the combined intervention, but cannot identify which individual change caused an effect. Repeatedly adjusting a candidate after seeing results also changes the question; record a new experiment and validate on fresh tasks. ## A result has a boundary The early-access comparison uses paired trials and verified task success as its primary outcome. It reports the candidate-minus-baseline difference with an uncertainty interval. A wide interval is useful information: the experiment has not resolved the question. A positive result on one task and model configuration does not establish a universal rule. Test additional tasks and agent versions before generalizing. Human safety and accessibility obligations also remain relevant when an agent metric improves. We apply the same standard to Evals AX. A detected issue should lead to a repair or an explained finding. An important missed issue should lead to a better detector or experiment, with a regression case showing the gap. --- # Run a local experiment Bring your agent runtime. Preserve the task, environment, and outcome needed to interpret each trial. ## The experiment contract An experiment manifest records a hypothesis, a task, baseline and candidate variants, the agent configuration, an independent verifier, and a fixed repetition count. The framework validates the manifest before execution. Keep the manifest in version control beside the environment it describes. ```sh evx experiment init ./my-experiment evx experiment validate ./my-experiment/manifest.json evx experiment run ./my-experiment/manifest.json --output ./results ``` Built-in Codex and Claude Code adapters are read-only in this release: Codex uses its read-only sandbox and Claude is limited to reading and searching files. An explicit custom command can integrate other trusted local runtimes. A fresh copied workspace provides repeatability; it does not isolate an arbitrary custom command from the host. Local execution requires macOS or Linux, including WSL. Native Windows execution stops before starting subprocesses or creating trial artifacts because this release cannot enforce Windows process-tree cleanup. Analysis, context inspection and account commands do not require local experiment execution. ## Verify the outcome independently The verifier runs outside the agent's workspace and returns a structured result. It should inspect the actual task outcome: for example, execute acceptance tests or check an artifact against a specification. Do not substitute the agent's confidence or final message for verification. ```json { "passed": true, "reason": "The output satisfies the task specification", "metrics": {} } ``` Independent process execution prevents accidental coupling; it does not make an arbitrary verifier correct or tamper-proof. Review what the verifier can observe, and test it against intentionally wrong outputs. The manifest supports positive and negative verifier controls. When configured, they run before the first agent trial; a verifier that accepts the wrong-answer control fails qualification. ## Read the comparison Our [API workflow experiment](/evidence/production-api-handoff-2026-09-08) shows eight real Codex trials using captured production responses. It also shows why verifier coverage matters: the original verifier rejected valid quotations. The recorded verdicts remain unchanged, with a separate assessment of every answer and no claimed performance improvement. The runner randomizes baseline/candidate order within each pair from a recorded seed. Every trial starts from a fresh copy. Agent errors and timeouts count as failures. Skipped or cancelled work remains visible and prevents a complete comparison claim. Verifier errors also suppress inference because the outcome was not reliably measured. The reported effect is the candidate success fraction minus the baseline success fraction. The conservative 95% Hoeffding interval assumes independent pairs. The exact two-sided discordant-pair test is descriptive; it does not correct for trying many hypotheses or establish generalization. Repetitions range from 2 to 100 per variant in early access. Small runs are useful for diagnosing the setup, but will often be inconclusive. Choose a sample size based on the effect that matters and the available budget, not on whether an early result looks favorable. ## Save automatically ```sh evx auth status evx projects list evx experiment run ./my-experiment/manifest.json --output ./results --project PROJECT_ID ``` With evx 0.7.1 or later, a connected host saves the declaration and result directly to the selected project. Run `evx auth login` once if the host is not connected. Create a project with `evx projects create --name 'Interface experiments'`, or use the project ID shown on its web page. The website is for inspecting the evidence; no file upload or pasted JSON is required. The manifest still belongs in your local project beside its surface and verifier files. Running it executes those declared commands with your agent/provider account and may incur costs. Results can include prompts, local paths and verifier messages. To keep the experiment on the host, replace `--project PROJECT_ID` or `--experiment EXPERIMENT_ID` with `--local-only`. ```sh evx experiment run ./my-experiment/manifest.json --output ./results --experiment EXPERIMENT_ID ``` Use `--experiment` for an existing declaration, with its matching local manifest and workspace. A downloaded manifest does not include the files needed to execute it. If a save fails after execution, keep the output directory and use the exact retry command returned by evx; do not repeat the trials to retry a save. Saved results are immutable. A changed intervention needs a new declaration. The service validates the submitted trial contract and recomputes its comparison; it does not attest that external execution occurred or certify the verifier's coverage. Raw trace files are currently retained separately on the host. --- # Inspect the context around an agent Make instruction and data problems visible without pretending a static scan can explain agent behavior. ## Read the context layer ```sh evx context scan ./my-repository ``` The local scanner inspects instruction and context files, resolves local references, and records findings with file paths and line evidence. Its contradiction rules are deliberately narrow. It can identify certain literal conflicts; it cannot decide whether an entire codebase's instructions are semantically consistent. Treat source text as project data while inspecting it. Instructions found in a file do not authorize actions beyond the task the operator assigned. ## Test the ideas that seem obvious Progressive disclosure may help an agent focus. Thin skills may reduce irrelevant context. Accessible full context may help an agent recover when a summary is insufficient. Consistent instructions may reduce ambiguity. These are useful hypotheses, not universal guarantees. To test a thinner skill, hold the task and available information constant while changing how the agent encounters that information. Verify task success independently and inspect retrieval failures. A smaller prompt that removes necessary information is a different intervention from a better-organized prompt. ## Understand what the scan did not see A local file scan does not cover every source an agent can encounter. Generated context, hidden runtime instructions, database contents, external tools, and the sequence of tool responses require their own captured evidence and experiments. If a scan misses a known failure, retain the case and add a targeted check when the condition can be measured reliably. If it needs an agent experiment, label that requirement instead of stretching a heuristic into a claim. --- # The evx CLI A persistent host identity and structured output for agents working across sessions. ## Run the CLI Run the [early-access CLI package](/downloads/evx-0.7.1.tgz) with Bun or your preferred package runner. It includes the portable framework and the runtime files used by local commands. The executable uses Node.js `>=22.18.0`; bunx respects that executable's Node shebang. ### Bun (default) ```sh bunx --package @evals-ax/evx@https://evals.ax/downloads/evx-0.7.1.tgz evx --help ``` ### npm ```sh npx --yes https://evals.ax/downloads/evx-0.7.1.tgz --help ``` ### pnpm ```sh pnpm dlx https://evals.ax/downloads/evx-0.7.1.tgz --help ``` ### Yarn 4 ```sh yarn dlx -p @evals-ax/evx@https://evals.ax/downloads/evx-0.7.1.tgz evx --help ``` For a persistent command on your host, install the same versioned package: ### Bun (default) ```sh bun add --global @evals-ax/evx@https://evals.ax/downloads/evx-0.7.1.tgz evx --version ``` ### npm ```sh npm install --global https://evals.ax/downloads/evx-0.7.1.tgz evx --version ``` ### pnpm ```sh pnpm add --global https://evals.ax/downloads/evx-0.7.1.tgz evx --version ``` Install `@evals-ax/evx`; run `evx`. These commands use the [versioned archive](/downloads/evx-0.7.1.tgz) because this version has not yet been verified on npm. The unrelated npm package named `evx` is not this product. New installations use `@evals-ax/evx`, with the command `evx`. Earlier package names are retained for compatibility. `@evals-ax/framework` is the MIT-licensed portable library included in this runtime. The hosted auditor and account-service implementation are not distributed in the CLI. Excluding source maps does not make the bundled framework private. Commands return JSON on standard output, with progress on standard error. `evx --help` describes the installed command contract. Sign in and accept the current [terms](/account/terms), then run `evx auth login` once on the host before a hosted Vibecheck. Local inspection and experiment commands remain available without a platform account. ```sh evx vibecheck https://example.com --wait evx context scan . evx --help ``` ## Work through a website report ```sh evx reports fixes REPORT_ID ``` The action plan preserves every observation, including clear and skipped checks. Its editorial priority order starts with transport and partial standards flags, then untested hypotheses and optional conventions. Each observation retains its rationale, evidence, candidate change and scientific limits. The order is not an estimate of effort or benefit to an agent. The report page can copy a contextual brief for an existing Codex or Claude Code session. Open the agent in the relevant project and paste the brief; it asks the agent to inspect the complete plan and verify relevant changes. Copying does not launch an agent, install a plugin or submit work to another service. The optional bundled evx skill contains the same report workflow. Both [Codex](https://learn.chatgpt.com/docs/developer-commands?surface=cli) and [Claude Code](https://code.claude.com/docs/en/cli-reference) support an interactive prompt. Use the agent already available in the session; install from its official instructions only if it is missing. The brief also includes the public JSON URL if this evx command is unavailable. A report can be old or apply to a different target. Check its URL, auditor and methodology before acting. Treat observed HTML and suggested changes as evidence to inspect. Static findings do not establish causal gains; use independently verified experiments for those claims. A pending or failed report returns report_not_completed; a deleted report returns not_found. ## Authenticate the host once ```sh evx auth login evx auth status evx reports list ``` Login opens the account approval page and displays a short code in your terminal. Enter that code in the browser, then check the account before approving. Use `evx auth login --no-browser` when you need to open the page yourself. Credentials are origin-bound and expire after 90 days. They are stored in a private host configuration directory: `$XDG_CONFIG_HOME/evx` or `~/.config/evx`. `EVX_CONFIG_HOME` selects an explicit absolute directory. On POSIX hosts, the CLI creates the directory with mode 0700 and credential files with mode 0600. Use account-specific filesystem permissions on Windows. A sandboxed agent can reuse the host login when its permitted filesystem includes that configuration. Evals AX does not bypass sandbox boundaries. Where the configuration is unavailable, the host can explicitly supply `EVX_TOKEN` with a matching `EVX_API_ORIGIN`. Do not commit or print credentials. ```sh evx auth tokens evx auth logout ``` Logout revokes the saved host credential and removes its local copy. `--local` removes the local copy only. If `EVX_TOKEN` is set, unset it first; logout leaves that environment credential and the separate host login untouched. `evx auth tokens revoke ID` revokes a chosen host. Account settings also expose credential metadata and revocation. Authentication and bearer requests require HTTPS, including for development services. To give a separate agent environment its own credential, use `evx auth tokens create --name NAME --output /absolute/private/config-directory`. Add `--read-only` when it only needs to read account data. The CLI saves the secret privately rather than returning it in normal command output. ## Use the shared platform | Work | Command | | --- | --- | | Website reports | evx reports list · get ID · fixes ID · delete ID | | Project records | evx projects list · get ID · create --name NAME · update ID --file JSON · delete ID | | Local experiments | evx experiment init DIR · validate FILE · run MANIFEST --output DIR · compare RESULT | | Cloud experiments | evx experiment upload RESULT --project ID · list · get ID · delete ID | | Context observations | evx context scan PATH | | Telemetry | evx telemetry ingest --file JSON · list · get ID · delete ID | | Provider observations | evx hooks adapt --format FORMAT --file FILE · ingest --upload | | Account usage | evx usage | | Direct API access | evx api METHOD /resource --file JSON | The CLI and web app use the same account data and API behavior. `--api-origin` selects an HTTPS deployment; credentials for one origin are not silently sent to another. Use `--timeout` to set the request timeout in milliseconds. List commands accept `--cursor` and return a page with `items` and `nextCursor`, which is null on the final page; they do not fetch every page automatically. `--file -` reads JSON from standard input. Failures emit structured JSON on standard error. Exit status is 0 for a successful command, 1 for a failure, and 130 for cancellation. A successful HTTP response can still contain a failed audit or an inconclusive experiment; inspect the domain status before deciding what to do next. [Provider adapters](/docs/telemetry#provider-events) inspect captured runtime output locally before an optional upload. [Agent integrations](/docs/integrations#install) include a thin skill and disabled-by-default hook examples in the same release package. --- # The API contract One account model and one set of operations across the web app and CLI. ## Read the machine-readable contract The versioned API begins at `/api/v1`. The [OpenAPI document](/api/v1/openapi.json) is the reference for request fields and responses. Use it instead of reconstructing an endpoint from a screenshot or this overview. Account operations accept an account session or a bearer credential issued through the device sign-in flow. Creating a report requires an account; existing public-link reports remain readable without signing in. Project and experiment data are owner-scoped. ## Resource overview | Resource | Purpose | | --- | --- | | /reports | Create a public website audit; list account reports. | | /reports/{id} | Read a public report or delete an owned report. | | /reports/{id}/events | Read recorded run stages as JSONL; resume with ?after=SEQUENCE. | | /reports/{id}/fixes | Read all observations with editorial priorities, evidence and an agent brief. | | /projects | Create and list private projects. | | /projects/{id} | Read, update, or delete an owned project. | | /experiments | Create and list experiment records. | | /experiments/{id} | Read a record, upload its immutable result, or delete it. | | /telemetry | Submit or inspect account telemetry. | | /usage | Read account usage and limits. | | /account | Inspect the authenticated account and terms status. | | /account/terms | Read or explicitly accept the current terms version. | | /auth/device | Begin a device authorization request. | | /auth/device/token | Poll an approved device request. | | /auth/tokens | List host credential metadata. | ## Submit and read a website audit Accept the current [terms](/account/terms) and authenticate with `evx auth login`. Save this request as `audit.json`: ```json { "url": "https://example.com" } ``` Submit through the CLI, which supplies the saved host credential to `POST /api/v1/reports`: ```sh evx api POST /reports --file audit.json ``` The response uses HTTP 202 when the audit has been accepted. This is an illustrative response shape, not a completed live report: ```json { "id": "00000000-0000-4000-8000-000000000001", "url": "/reports/00000000-0000-4000-8000-000000000001", "visibility": "public-link", "savedToAccount": true } ``` Read `GET /api/v1/reports/{id}` to inspect `status`. It can be queued, running, completed, or failed. A completed run contains its report; a failed run contains an error. The CLI's `--wait` option handles polling. The report's `progress` array records actual service stages and available counts. `GET /api/v1/reports/{id}/events?after=2` returns a bounded JSONL snapshot after sequence 2; poll until `X-Evx-Run-Status` is completed or failed. These events have the report's public-link visibility. Older reports can have no events. Static runs do not include model trials or sandbox execution. A project creation uses `POST /api/v1/projects` with `name` and an optional `description`. It returns HTTP 201 with the project's ID, normalized fields, and creation timestamp. Collection reads return `items` and `nextCursor`; pass a non-null cursor back as `?cursor=VALUE` to read the next page. ## Review terms before account writes New account work and host credentials require explicit acceptance of the [terms](/terms). Existing accounts are prompted too; earlier use is not recorded as agreement. Reads, exports and deletion remain available. New audits require a signed-in account and current terms acceptance. Read `evx api GET /account/terms`. After reviewing the returned version and obtaining authority to agree for the account, save `{ "version": "2026-09-08", "accepted": true }` to `acceptance.json` and send it using the command below. This example version must match the current response. Do not automatically accept on behalf of someone who has not authorized you. ```sh evx api PUT /account/terms --file acceptance.json ``` A required agreement returns HTTP 428 with `terms_required` and review links. A stale version returns HTTP 409 with `terms_version_changed`; read and review the new terms before making a new acceptance request. Retrying the same current acceptance keeps its first timestamp. Browser users can [review account terms](/account/terms). New CLI connections offer the same acceptance form inline at `/connect`, retaining the entered device code while you review the policy in a separate tab. ## Handle errors as data ```json { "error": { "code": "sign_in_required", "message": "Sign in to use your account data." }, "requestId": "00000000-0000-4000-8000-000000000001" } ``` Failures include a stable code, a readable message, and a request identifier. A rate-limited response can include `retryAfterSeconds`. Inspect the status and error code before retrying; changing credentials will not fix an invalid request. Keep credentials in authorization headers. Do not put bearer tokens in URLs, which can become part of browser history or logs. See [CLI authentication](/docs/cli#authentication) for host credential storage and revocation. --- # Measure effort without inventing data Record what the runtime exposes and preserve what remains unknown. ## A measurement needs provenance Model identifiers, reasoning settings, token usage, elapsed time, and monetary cost help explain an outcome only when their origin is known. Runtime-reported usage and externally supplied telemetry are different evidence sources. Missing usage is unknown, not zero. The experiment record combines independently verified success with available execution measurements. A hook can report an event or interruption, but a label such as confusion or intent misalignment needs an operational definition before it becomes a comparable metric. ```sh evx telemetry ingest --file event.json evx telemetry list ``` ## Inspect provider events before sharing them `evx hooks adapt` reads a captured provider JSON or JSONL file and returns the normalized events with explicit coverage. It makes no network request. Generate one run ID for a capture and retain it for retries; formats without stable event identifiers require it. ```sh RUN_ID=$(node -p 'require("node:crypto").randomUUID()') evx hooks adapt --format codex-jsonl --file codex.jsonl --run-id "$RUN_ID" ``` | Format | Available observations | | --- | --- | | codex-jsonl | Completed-turn token usage; --run-id required. Model, elapsed time, and cost remain unknown. | | codex-notify | agent-turn-complete lifecycle event; no token or cost measurements. | | claude-result | One terminal result: main-agent token usage, elapsed time, and client-estimated invocation cost including subagents. Model only when one model is reported. | | claude-hook | SessionStart, Stop, or StopFailure; --run-id required. SessionStart may report a model; no usage measurements. | Every adapter leaves task outcome unverified and reasoning level unknown. Provider completion is not an independent task verifier. The Claude result adapter requires exactly one result to avoid summing ambiguous cumulative costs. These are format adapters; they do not establish continuous runtime coverage, and the Claude integrations have not been qualified with a live Claude session. Prompts, tool arguments, response text, raw errors, working directories, and transcript paths are discarded. The adapter never opens a referenced transcript automatically. Only the selected measurements and bounded provenance fields enter the event record. After inspecting the local output, an authenticated host can explicitly upload the same capture. Keep the run ID unchanged for a retry. `--file -` also accepts standard input. ```sh evx hooks ingest --upload --format codex-jsonl --file codex.jsonl --run-id "$RUN_ID" evx telemetry list ``` ## Separate measured cost from an estimate A token-derived estimate depends on the exact model, provider pricing, and any cache or batch discounts. An invoice includes costs that token counts may omit. Store the estimate's assumptions rather than presenting it as an observed bill. Local agent execution uses your runtime and provider account. Early-access cloud storage does not run a model on your behalf. Website auditing still consumes network, function, and database resources even when no model is involved. ## Bound the experiment Set execution timeouts and the repetition count before running. Platform admission limits bound public audit throughput. Operational controls can pause new work when resource usage requires intervention; a pause should be visible as an explicit error rather than a silently missing result. Share the smallest record that supports the analysis. Review traces for credentials and private task data before uploading. Aggregate costs without discarding the identifiers needed to investigate an outlier. ## Early-access storage limits Each account can store 50 projects and 200 experiment records, with a total storage limit of 50 MB and a 2 MB limit per result upload. The usage API reports the account's consumption. Telemetry events are retained for 30 days. Anonymous public report links expire after 7 days. A daily cleanup removes expired data within a further 24 hours. Save work to an account when you need persistent ownership, and keep local experiment artifacts as your original record. --- # What our first study found All tasks succeeded. The comparison stayed inconclusive, while the traces exposed work that the success metric missed. ## Would a direct source index help? We asked Codex to recover exact facts about Evals AX authentication from a selected snapshot of our shipped code and architecture documentation. The candidate added an instruction-file index pointing to canonical implementation files. Both conditions contained the same underlying source and documentation. The local plan was recorded at `2026-09-07T08:14:18.471Z`, before execution began at `2026-09-07T08:14:18.474Z`. Verified task success was the primary outcome. Duration and provider-reported usage were declared secondary descriptive measurements. This is a locally recorded plan, not independent third-party registration. | Condition | Recorded configuration | | --- | --- | | Requested agent | codex / gpt-6-astra / low reasoning | | Runtime | codex-cli 0.153.0-alpha.5 | | Framework | 0.1.0 | | Design | 12 pairs, randomized order within each pair; seed 20260907 | | Task version | 69ab3276-lookup-v1 | | Trial timeout | 45.00 s | | Verifier | auth-facts-v1-with-positive-and-negative-controls | ## The success comparison was inconclusive | Primary measure | Baseline | With source index | | --- | --- | --- | | Verified successes | 12 / 12 | 12 / 12 | | Uncompleted trials | 0 | 0 | The observed candidate-minus-baseline success difference was 0 percentage points. The conservative 95% paired Hoeffding interval ranged from -78.4 to +78.4 percentage points, assuming independent pairs. Every trial succeeded, leaving this task at a ceiling for the chosen success criterion. The result does not establish equivalence, demonstrate an improvement, or show that an index is unnecessary. A harder or more varied task set is needed to examine success differences. The confidence calculation assumes independent repeated pairs. Fresh workspaces did not reset provider caches or eliminate service drift; those are limits on interpreting the interval. ## Other measurements are descriptive | Observation | Baseline | With source index | | --- | --- | --- | | Mean agent duration | 25.75 s | 32.89 s | | Reported input tokens, total | 641,053 | 835,919 | | Reported output tokens, total | 2,439 | 2,867 | | Trials reporting dollar cost | 0 / 12 | 0 / 12 | No inferential comparison of efficiency was planned. The timing and token totals do not establish that a source index makes agents slower or more expensive. Dollar cost is unknown, not zero; token counts are not a provider invoice. In the candidate condition, each trial attempted an API route path absent from the selected snapshot and recovered through search. The index linked to files present in the snapshot; the agent inferred the missing route paths from the source structure. A separate post hoc extraction counted 24 completed commands and 0 nonzero exits in the baseline, compared with 36 commands and 19 nonzero exits in the candidate. Those candidate exits included missing-file reads and searches with no matches; a nonzero exit is not automatically an error. That snapshot limitation matters. A failure to read an absent route in a selected snapshot is not evidence about the same task in the full repository, and the added index itself had no broken links. ## A passing task can hide recovery work The final success metric missed the intermediate failed reads because the agent recovered and returned correct answers. We added provider-event diagnostics that retain command counts and nonzero exit positions without copying raw command text into uploaded results. Adapters without diagnostic coverage report unknown values. The original result remains unchanged. The later diagnostic extraction is recorded separately, rather than presented as a measurement captured by the original runner. A failed tool command is still an observation; calling it confusion or intent misalignment requires further evidence. ## Inspect the public record The [aggregate study record](/evidence/auth-context-2026-09-07.json) contains the configuration, source revision identifiers, manifest hash, and numerical comparison. It is derived from the framework's retained result and summary. Source snapshots and raw traces are not included in that public response. Manifest hash: `cc6afc0f7888bfcbd201fa93b70f18c11289f55d34591a7add6bce2dfe646fef`. This was a selected authentication-source snapshot, not the full repository. The observations do not establish that indexes help or harm agents in other environments. Local plan timestamps and client-produced evidence are not independent third-party attestation. --- # Integrate without duplicating the rules Keep the domain contract shared. Make harness-specific adapters small. ## Give an agent a clear entry point An agent can start with `evx --help`, read the relevant documentation, and call the shared API through the CLI. [Full documentation as Markdown](/docs/full.md) remains available when a short entry point is insufficient. A skill should explain when the capability is useful and where its authoritative contract lives. A hook should translate a runtime event into the shared telemetry schema. Neither should maintain its own version of the experiment method or claim that an observed behavior is universally better. ## Use the bundled skill or plugin The [versioned CLI package](/downloads/evx-0.7.1.tgz) includes `integrations/evals-ax`. After installing with your preferred package manager, run `evx integrations path` to find its absolute path. This works for Bun installations as well as npm and the other runners. The integration contains `skills/evx/SKILL.md` and manifests for Codex and Claude Code. Install the skill through your host's skill installer, or load the directory through its plugin installation interface. The skill points to `evx --help` and these docs; it does not embed a second copy of the method. For one Claude Code session, use its [local plugin loader](https://code.claude.com/docs/en/plugins-reference#debugging-and-development-tools): ```sh evx integrations path claude --plugin-dir "/absolute/path/returned/by/evx" ``` The release includes both plugin manifests, but does not install anything into your agent host automatically. A bundled README records supported formats and qualification limits. Installing the skill or plugin does not enable telemetry. The `examples` directory contains optional host configuration snippets; preserve existing configuration when applying them. Codex accepts one notification command, so integrate with an existing command before changing that setting. ## Enable lifecycle observations deliberately The bundled `scripts/observe.mjs` accepts the documented Codex notification or Claude lifecycle event. It stays inactive unless the host sets `EVX_HOOKS_UPLOAD=1` and `EVX_BIN` to the absolute path of an installed, authenticated `evx` executable. The example configuration files show where to connect that script in each host. The wrapper uploads only the allowlisted [provider event fields](/docs/telemetry#provider-events). It produces no standard output or agent decision fields, returns exit status zero, and limits observation delivery to five seconds. Failed delivery is reported generically on standard error, with no automatic retry. This is best-effort telemetry, so an absent event is not evidence that no work occurred. Lifecycle hooks do not report tokens or costs. Capture explicit provider result output when those measurements are needed, and use an independently verified experiment to assess whether a change improved the intended outcome. ## Ask CodeRabbit to review AX effects The [AX review template](/templates/coderabbit-ax.yaml) is a small CodeRabbit configuration derived from the thin collaborator approach. It asks the reviewer for concrete evidence about discoverability, instruction consistency, and behavioral changes. It does not turn preferences about agent interfaces into hard rules. Copy the relevant guidance into an existing `.coderabbit.yaml`, preserving your repository's settings. Installing the file does not install the CodeRabbit GitHub app; enable repository access separately. A useful finding names the affected agent task and the code or contract behind the concern. When an effect is uncertain, propose a test. Investigate disagreements with the reviewer rather than treating approval as a substitute for evidence.