Decode's harness verifies coding agents with hidden tests in fresh sandboxes

• Decode's evaluation harness hands back a branch, then grades it with a verifier the agent never sees, according to the seven-step design.
• Every trial starts from a fresh repository, runs inside Docker or Modal, and applies hidden tests to a pristine clone, as the verifier workflow shows.
• The course separates “does it work?”, “does it still work?”, and “does it keep working?” into benchmarks, regression tests, and online evals, per the three-signal model.
• The sharp trust boundary is simple: the agent produces the patch, while the six-layer architecture map puts grading outside the core agent loop.

Decode's open-source course repository reduces the tool-calling agent to roughly 20 lines, then spends the rest of the system on tools, permissions, sandboxes, memory, runtime, and evals. The evaluation guide names the concrete machinery: hidden tests/test.sh graders, reward files, trial directories, regression gates, and live-trace mining. A separate patch-scoring schema reaches the same engineering conclusion: score the returned diff and hidden constraints, not the agent's success narration.

The failure that motivated Decode's harness was a one-commit Git task. The agent said the newest commit had been reverted and that all tests passed, but the commit was still present, according to the failure report.

The returned message described success; the repository state contradicted it. the follow-up failure example makes the distinction explicit: the verifier checks the environment after the agent finishes and returns either 1 or 0.

The design's sharpest choice is its trust boundary. The agent can propose and execute a patch, but it does not get to decide whether the patch worked.

Decode's trial pipeline is deliberately repetitive:

1. Create a fresh repository from the same starting state.
2. Run the agent in an isolated Docker or Modal sandbox.
3. Collect the branch the agent hands back.
4. Clone that branch into a pristine environment.
5. Add hidden tests the agent has not seen.
6. Run the tests and assign a binary score.
7. Record the result and debugging evidence.

The repository's isolated-workspace ADR says the harness, not the model, owns the hand-back: it collects the workspace state and points a deterministic session branch at it. The current eval guide describes one trial as a decode run subprocess whose grader runs host-side on a pristine clone.

The agent therefore sees its task environment and its own tool results, but not the final test script. That removes the easiest path to a benchmark win based on reading or gaming the grader.

The benchmark is meant to predict whether the agent can complete a real task in a controlled environment, not whether it can produce a convincing completion message. The current benchmark implementation specifies:

• 19 Terminal-Bench-style tasks: 7 easy, 6 medium, and 6 hard.
• A hidden tests/test.sh script that writes one reward to reward.txt after hand-back.
• Repeated trials, where --trials 3 produces pass@3, pass^3, and a flakiness value.
• Separate accounting for infrastructure failures, which are excluded from the agent score rather than counted as model losses.

The distinction between pass@k and pass^k captures capability versus consistency. One successful run can establish that an agent sometimes solves a task; repeated successful runs test whether it does so reliably.

The course frames evaluation as three questions: “Does it work?” for benchmarks, “Does it still work?” for regression tests, and “Does it keep working?” for online evals, as the three-signal split puts it.

Decode's current implementation turns the second and third questions into concrete machinery:

• 21 regression cases, divided into 5 easy, 8 medium, and 8 hard cases.
• A deterministic dataset surface with threshold gates for exact behavior such as tool discipline and judge scores.
• An English-assertion test suite scored by an LLM judge, for regressions that a single numeric metric may miss.
• A mine command that searches live Opik traces, clusters failures, and turns selected traces into new regression cases.

The regression guide treats live traffic as a source of future tests rather than as a detached dashboard. A production failure can become a versioned case with its source trace attached, then enter the same gate as hand-written cases.

The evaluation plane sits below a larger harness, not inside the model call. The architecture separates:

• Surface: CLI, SDK, IDE, approvals, and steering.
• State: conversation history, project instructions, sessions, and memory.
• Core: the model call, tool dispatch, and observed result loop.
• Safety: automatic execution, permission requests, and blocked actions.
• Backend: local or remote execution, tools, MCP servers, and sandboxes.
• Evals: benchmarks, regressions, and production monitoring.

The broader architecture recap adds compaction and skills to the state around that loop. The terminal interface also splits mid-turn input into a steering queue and a follow-up queue, according to the two-queue design, so a user can influence the current turn without confusing it with work that should wait for the next turn.

Remote execution makes the same boundary physical. Decode's repository documents disposable Modal sandboxes, while a cloud-agent CLI example shows the adjacent pattern in practice: SSH into a cloud agent, run tests in a real VM and browser or device environment, and return a test result to the operator.