Evaluating AI coding agents

The problem

A deterministic system has one correct output per input, so a test is an equality check. An agent has a set of correct outputs. Two agents can solve the same task by different routes, with different intermediate commands, different file layouts and different orderings, and both be right. Assert on the transcript and you fail correct work. Assert on nothing and you are measuring vibes.

The second problem is worse, because it is invisible. A failed evaluation is ambiguous by default: when a run goes red, the honest question is whether the agent was wrong or the environment was — a flaky network call, a file leaked from a previous trial, a clock, a version that floated. Until that ambiguity is engineered out, the score is a measurement of your harness rather than of the model.

How it works

The unit of work is a task, and a task is four things that ship together: a written specification, a Dockerised environment, a reference implementation, and an automated verification suite that defines objective pass/fail. Miss any one of them and what you have is not an evaluation, it is an anecdote.

The reference implementation is not documentation. It is the control. If the verifier does not pass against a known-good solution, the verifier is wrong — and you find that out before an agent's failure gets misattributed to the agent.

task/
├── spec.md            what the agent is asked to do, in prose,
│                      the way a ticket would actually arrive
│
├── environment/
│   └── Dockerfile     pinned base image, pinned dependencies,
│                      clean container per trial,
│                      no network access at solve time
│
├── solution/          reference implementation — proof the task
│                      is solvable, and the control the verifier
│                      is run against first
│
└── tests/             the verifier
                       asserts on outcomes and artifacts
                       never on the transcript or the route
                       deterministic, or it does not ship

Decisions worth defending

1. 01

   Assert on outcomes, never on the route.

   The verifier checks what is true once the agent has stopped — the endpoint returns the right shape, the file has the right contents, the process exits clean, the migration applied. Not which commands got it there. That is the whole trick that lets two valid approaches both pass while a wrong outcome still fails, and it is the direct analogue of testing behaviour instead of implementation.

2. 02

   Determinism is a property you build, not one you hope for.

   Pinned base images, pinned dependency versions, no network at solve time, a clean container per trial, and no state carried between runs. Every source of variance that is not the agent gets removed. The payoff is narrow and total: a red run means one thing.

3. 03

   Failure modes are the design target, not a side effect.

   Boundary, negative and exploratory cases are aimed at where agents actually break — long-horizon planning that loses the thread halfway, environment state the agent assumed instead of checked, and recovery after a tool call fails. A task that only exercises the happy path tells you nothing you did not already know about a model that was trained on happy paths.

4. 04

   A flaky verifier is a broken verifier.

   A test that fails one run in twenty is worse than no test, because it launders environment noise into a capability measurement and does it with a straight face. Flakes get fixed or the task gets pulled. They do not get re-run until green.

5. 05

   Difficulty is calibrated against the reference.

   A task every agent trivially passes and a task nothing can pass carry the same amount of signal: none. The reference implementation is the yardstick for how much work a task actually is, which is what lets a set spread across difficulty instead of quietly clustering at whatever was easy to author.

Proving it works

What carries over from deterministic testing, and what does not.

• Carries over: boundary analysis, negative cases, equivalence partitioning — the discipline of asking what the input space actually looks like before writing a single assertion.
• Carries over: treating the test environment as production code. Pinned, reproducible, reviewed, and owned.
• Carries over: distrusting a green run you have never watched fail. A test that has not been proved capable of failing is not evidence.
• Does not carry over: exact-match assertions. Correctness is a set, not a value.
• Does not carry over: a single run as evidence. When the system under test is probabilistic, the pass/fail criterion has to be stable across runs before any individual result means anything.
• New: the reference implementation as a control on the verifier — the closest thing this discipline has to a unit test for the test.