Evaluating AI coding agents

The problem

So the first thing you asked is why agents are hard to test. With normal software one input gives one correct output. That makes a test a simple check that two values are equal. An agent can give many correct outputs. Two agents can solve the same task in different ways with different commands and different files and both can be right. If I check the transcript I fail good work. If I check nothing I only measure a feeling.

The second problem is worse to me because you cannot see it. When a run fails you do not know at first who is wrong. It can be the agent or it can be the environment. Maybe a network call was flaky or a file stayed behind from the last trial. Maybe the clock was off or a version moved. If I do not remove these things the score tells you about my setup and not about the model.

How it works

You also asked what I actually make. I make tasks and each task has four parts which always go together. There is a written spec and a Docker environment. There is a reference solution and a set of automated tests that decide pass or fail. If one part is missing it is not an evaluation. It is only a story.

The reference solution is not there as documentation. It is the control. I run the tests against a solution that I know is good. If the tests fail on it then the tests are wrong and not the agent. I find that out before anyone blames the agent for my mistake.

task/
├── spec.md            what the agent must do in plain words
│                      the way a real ticket would read
│
├── environment/
│   └── Dockerfile     pinned base image
│                      pinned dependencies
│                      a clean container for each trial
│                      no network while the agent works
│
├── solution/          the reference solution which shows
│                      the task can be solved and which
│                      the tests run against first
│
└── tests/             the verifier
                       checks the result and the files
                       never the transcript or the route
                       same result every run or it does not ship

Decisions worth defending

1. 01

   How do I check the work when there is more than one right way?

   I check the result and not the steps. When the agent stops my tests look at what is true. Does the endpoint give back the right shape? Does the file have the right content? Did the process exit clean and did the migration run? I do not care which commands got it there. This lets two good approaches both pass and a wrong result still fails. It is the same idea as when you test behaviour and not the implementation.

2. 02

   How do I make the results the same every time?

   I do not hope for it. I build it. I pin the base image and the version of every dependency. The agent gets no network while it works. Each trial gets a clean container and nothing carries over from the last run. I take out every cause of change that is not the agent. So when a run fails it can only mean one thing.

3. 03

   What kind of cases do I write?

   I aim at the places where agents really break. One is a long task where the agent loses track halfway through. Another is when the agent guesses the state of the environment and does not check it. The third is how it recovers when a tool call fails. I write boundary and negative and exploratory cases for these. A task that only tests the happy path tells you nothing new because the model was already trained on happy paths.

4. 04

   What do I do with a flaky test?

   To me a flaky test is a broken test. If a test fails one run in twenty it is worse than no test. It turns noise from the environment into a score and the score looks real. So I fix the flake or I pull the task out. I do not run it again and again until it goes green.

5. 05

   How do I know a task is not too easy or too hard?

   I use the reference solution to measure it. If every agent passes a task with no effort it tells you nothing. If no agent can pass it that also tells you nothing. The reference shows how much work the task really is. That way my set has easy and hard tasks and does not all sit at the easy end which is what happens when you only write what is quick to write.

Proving it works

The last thing you asked was what still works from normal testing and what does not. Here is my honest list.

• Still works: boundary analysis and negative cases and equivalence partitioning. I still ask what the inputs really look like before I write a single check.
• Still works: I treat the test environment like production code. It is pinned and you can reproduce it. Someone reviews it and someone owns it.
• Still works: I do not trust a green run that I never saw fail. If a test has not shown that it can fail it is not proof of anything.
• Does not work: exact match checks. There is a set of right answers and not only one value.
• Does not work: one run as proof. The agent does not give the same output each time so the pass or fail rule must stay stable across many runs. Only then does one result mean something.
• New for me: the reference solution as a control on the tests. It is the closest thing I have to a unit test for the test.