Building the benchmark

Notes on how the benchmark is put together, written as it happens.

• Oct 14, 2026Tasks a strong agent cannot skimFour runs of Claude Code on the Track N tasks, why the first three told us nothing about robustness, and what it took to get a task that needs more than a handful of tool calls.
• Oct 11, 2026Three harnesses pass conformanceThe conformance rerun for Claude Code, Pi and OpenCode, what each harness does when the injector breaks, and the three tasks the smoke run will use.
• Oct 10, 2026Does the fault reach the model?How faults get into a coding agent, why we test delivery before we test agents, and what the first conformance batch found.