What counts as evidence

A finding is a question worth testing

An observation describes something the system measured: a control has no accessible name, an instruction file links to a missing path, or a verifier rejected an agent's output. A hypothesis proposes an explanation. An experiment estimates the effect of a specified change under recorded conditions.

These are different evidence types. Static audits can find defects and suggest experiments. They do not establish a causal relationship between a design pattern and agent success.

Define the comparison before running it

1. Write the task and the observable success criterion. Use a verifier independent of the agent's own claim of completion.
2. Record the agent configuration and environment versions. Set the repetitions and budgets before inspecting outcomes.
3. Change one proposed cause where practical. Keep unrelated task conditions comparable and reset the environment for each trial.
4. Keep failures and incomplete runs in the record. Compare the planned sample, report uncertainty, and inspect what happened.

Changing several things at once can test the combined intervention, but cannot identify which individual change caused an effect. Repeatedly adjusting a candidate after seeing results also changes the question; record a new experiment and validate on fresh tasks.

A result has a boundary

The early-access comparison uses paired trials and verified task success as its primary outcome. It reports the candidate-minus-baseline difference with an uncertainty interval. A wide interval is useful information: the experiment has not resolved the question.

A positive result on one task and model configuration does not establish a universal rule. Test additional tasks and agent versions before generalizing. Human safety and accessibility obligations also remain relevant when an agent metric improves.

We apply the same standard to Evals AX. A detected issue should lead to a repair or an explained finding. An important missed issue should lead to a better detector or experiment, with a regression case showing the gap.