{"id":"evals-auth-context-index-2026-09-07","evidenceOrigin":"client-reported","planRecordedAt":"2026-09-07T08:14:18.471Z","manifestHash":"cc6afc0f7888bfcbd201fa93b70f18c11289f55d34591a7add6bce2dfe646fef","startedAt":"2026-09-07T08:14:18.474Z","finishedAt":"2026-09-07T08:26:03.203Z","frameworkVersion":"0.1.0","agent":{"kind":"codex","requestedModel":"gpt-6-astra","reasoning":"low","configuration":"isolated","runtimeVersion":"codex-cli 0.153.0-alpha.5"},"task":{"id":"discover-shipped-auth-contract","version":"69ab3276-lookup-v1","description":"Recover the exact shipped authentication contract from a selected source snapshot and its documentation."},"surfaces":{"baselineRevision":"website-69ab3276b7cbf6ec6ef28e2732c372a08aa29232-selected-auth-source","candidateRevision":"website-69ab3276b7cbf6ec6ef28e2732c372a08aa29232-selected-auth-source-plus-index-v1"},"design":{"repetitions":12,"seed":20260907,"minimumEffect":0},"limits":{"trialTimeoutMs":45000,"totalTimeoutMs":900000},"verifierVersion":"auth-facts-v1-with-positive-and-negative-controls","summary":{"evidenceOrigin":"client-reported","primaryOutcome":"independently-verified-task-success","baseline":{"planned":12,"attempted":12,"passed":12,"successRate":1,"failures":0,"errors":0,"notCompleted":0,"meanAgentDurationMs":25748.73594441666,"inputTokens":{"knownTotal":641053,"reportedTrials":12,"totalTrials":12,"complete":true},"outputTokens":{"knownTotal":2439,"reportedTrials":12,"totalTrials":12,"complete":true},"costUsd":{"knownTotal":0,"reportedTrials":0,"totalTrials":12,"complete":false}},"candidate":{"planned":12,"attempted":12,"passed":12,"successRate":1,"failures":0,"errors":0,"notCompleted":0,"meanAgentDurationMs":32889.449826166645,"inputTokens":{"knownTotal":835919,"reportedTrials":12,"totalTrials":12,"complete":true},"outputTokens":{"knownTotal":2867,"reportedTrials":12,"totalTrials":12,"complete":true},"costUsd":{"knownTotal":0,"reportedTrials":0,"totalTrials":12,"complete":false}},"effect":{"estimate":0,"confidence":0.95,"interval":[-0.7841002756996853,0.7841002756996853],"method":"paired-hoeffding","units":"success-rate-difference"},"discordance":{"candidateWins":0,"baselineWins":0,"pValue":1,"method":"exact-two-sided-binomial"},"conclusion":"inconclusive","scope":"This task, surface snapshots, agent configuration, verifier and execution environment only.","cautions":["The confidence bound assumes independent repeated pairs. Shared caches, provider drift or side effects can violate that assumption.","A verifier is independent of the agent's self-assessment, but its correctness and coverage require separate validation.","The p-value is descriptive for this predeclared comparison. Repeated peeking, selecting tasks or testing multiple changes needs a separate multiplicity plan.","Durations and usage are descriptive; missing token or cost telemetry is unknown, not zero."]},"postHocDiagnostics":{"derivedAt":"2026-09-07T08:36:47.326Z","interpretation":"Post hoc trace diagnostics. Every candidate attempted an API route file omitted from this selected-source snapshot, then recovered. Nineteen nonzero exits comprise twelve missing-file commands and seven searches returning no matches. Nonzero exit is not synonymous with a tool error or agent confusion. The original experiment result remains unchanged.","baseline":{"trials":12,"commandCalls":24,"nonzeroCommandExits":0},"candidate":{"trials":12,"commandCalls":36,"nonzeroCommandExits":19}},"interpretation":{"primaryConclusion":"Inconclusive. Every trial passed, so this task provides no observed success-rate separation between the conditions.","descriptiveOnly":"Duration and reported token usage were declared secondary descriptive measures. No efficiency inference was planned.","traceFinding":"In the candidate condition, each trial attempted an API route path absent from the selected snapshot and recovered through search. The index linked to files present in the snapshot; the agent inferred the missing route paths from the source structure.","limits":"This was a selected authentication-source snapshot, not the full repository. The observations do not establish that indexes help or harm agents in other environments. Local plan timestamps and client-produced evidence are not independent third-party attestation."}}