API, CLI and MCP inspections

Start with an explicit plan

Write a JSON plan, then run evx inspect. With a CLI login, the completed report saves privately to your account automatically; without one, it stays local. Add --local-only to opt out for any run. Reports can include interface names and diagnostic evidence. The CLI executes only the probes you select. An inspection is a set of interface observations, not an agent benchmark or a universal readiness score.

evx inspect ./inspection.json --output ./report.json

The plan bounds repetitions, per-operation time, total time and output size. Review commands before running them: starting a local program can have side effects even when a probe is named help. No local commands are executed by the hosted service.

Inspect an API

The API collector reads an OpenAPI JSON document and inventories its operations and schema descriptions. Optional GET or HEAD probes must be selected explicitly and remain on the configured origin. This example checks that an anonymous account request is rejected; it does not create an account or an audit.

{
  "schemaVersion": "1",
  "kind": "api",
  "name": "Evals AX public API",
  "target": {
    "url": "https://evals.ax/api/v1/openapi.json"
  },
  "probes": [
    {
      "path": "/api/v1/account",
      "method": "GET",
      "expectedStatuses": [
        401
      ]
    }
  ]
}

A successful HTTP response does not establish that an agent can complete an API workflow. Check response meaning with an independent verifier in an experiment when that is the question. Authentication headers can reference environment variable names through headersFromEnv; credential values do not belong in the plan.

Inspect a CLI

Choose exact argument arrays and expected exit codes. Processes run without a shell. The collector records bounded structural observations and output hashes; it does not upload raw process output by default. This example requires evx to be installed on PATH.

{
  "schemaVersion": "1",
  "kind": "cli",
  "name": "evx command contract",
  "probes": [
    {
      "id": "help",
      "purpose": "help",
      "command": [
        "evx",
        "--help"
      ],
      "expectedExitCodes": [
        0
      ]
    },
    {
      "id": "version",
      "purpose": "version",
      "command": [
        "evx",
        "--version"
      ],
      "expectedExitCodes": [
        0
      ]
    },
    {
      "id": "invalid-argument",
      "purpose": "invalid-input",
      "command": [
        "evx",
        "--not-a-real-option"
      ],
      "expectedExitCodes": [
        1
      ]
    }
  ]
}

A help flag and a nonzero invalid-input exit code are observations. They do not prove instructions are understandable or that the command is safe or useful. Use custom probes for the contract your tool actually promises.

CLI probes and MCP stdio transports can name host variables in env; their values are passed to the process and redacted from report text as credentials. Use publicEnv only for nonsecret settings such as NO_COLOR. Public values are not redacted, so they must never contain credentials. The plan stores variable names, not their values.

Inspect an MCP server

MCP inspections discover the server and list tool definitions without calling its tools. Set the exact protocol revision your server supports. The collector supports 2026-07-28 discovery and the older 2025-11-25 initialization flow; it does not silently downgrade. Replace the command below with a server you intend to start.

{
  "schemaVersion": "1",
  "kind": "mcp",
  "name": "My MCP server",
  "protocolVersion": "2026-07-28",
  "transport": {
    "type": "stdio",
    "command": [
      "node",
      "./server.mjs"
    ]
  },
  "maxPages": 5
}

HTTP transport uses type http with url and optional headersFromEnv in place of command. Pagination is bounded by maxPages. Missing descriptions and schema observations are leads for investigation, not demonstrated causes of agent failure. See the MCP versioning specification.

Save and inspect the report

evx auth login
evx inspect ./inspection.json --output ./report.json --title "Production API discovery"
evx interfaces list

After CLI sign-in, inspections appear in API, CLI & MCP reports as part of the run. The CLI writes and flushes the local file first, then saves through the shared API. Its JSON response includes cloud.status and cloud.reportUrl; saved confirms the account accepted the report, while local-only identifies an explicit opt-out or missing CLI login. Reports remain private and immutable. The service validates the contract and derives the displayed counts; it does not attest that host execution occurred.

For sensitive interfaces, use --local-only to review the report before sharing it. Retained reports count toward the account storage allowance and can be deleted. Raw local traces remain on the host. Experiment uploads and provider telemetry still require their separate explicit commands.

A save failure exits with code 1 and INSPECTION_SAVE_FAILED on stderr, keeping the local report. The error includes its path, report ID, safe reason and a retry command with an argument array. A network failure leaves cloud delivery unconfirmed; retrying the same retained report and title is idempotent. No inspection is rerun and no write is retried automatically. A 428 response links to the current terms for actual account acceptance; the CLI cannot accept them for you. Cancellation retains local evidence and exits 130.