Kitaru replays agent failures without repeating state-changing tool calls

• A coding agent’s provider fallback turned an expected $4 to $5 benchmark run into a $40 bill, as the developer described.
• Kitaru reconstructs failure context from recorded tool outputs without repeating matching API or database calls, according to the developer’s replay setup.
• Production failures become regression cohorts ranked by frequency × severity in the developer’s workflow.

The Decode walkthrough includes a 30-run seed set with 16 deliberately injected failures. Kitaru’s replay comparisons expose changes in spending and call counts alongside evaluator results.

Paul Iusztin, author of Building a Coding Agent From Scratch, was using Claude Code to implement Decode’s eval layer. A Modal endpoint returned HTTP 503 during warm-up; Claude Code then used a Gemini API key it found in memory, according to his account.

His walkthrough describes a two-line fix: wait for warm-up and guard credentials in environment variables. Kitaru, Modal, and Opik sponsor the course.

Kitaru re-executes the agent from the top, using recorded outputs for matching tool calls. Calls with different arguments, or tools changed during a fix, can execute afresh in the Decode walkthrough.

The default is a footgun: Kitaru’s replay guide says omitting the tool policy currently passes calls through to live tools. Its example is blunt: replaying refund_payment refunds the card again.

The documented fail-closed policy is:

This confines tool responses to the baseline’s recorded history and fails the replay when a call has no recorded match. Iusztin made that safety condition explicit in a reply:

Decode’s evaluation setup separates four roles:

1. Decode: the coding agent under test.
2. Control plane: records sessions and coordinates error analysis.
3. Workers: replay sessions in isolated environments on the developer’s infrastructure.
4. Investigating coding agent: analyzes failures through Kitaru’s MCP server and skills.

Iusztin’s walkthrough turns error analysis into six steps:

1. Record sessions: retain the goal, outcome, model calls, and tool calls.
2. Review: assign pass/fail labels with a short human critique.
3. Fix obvious errors: address failures that need no further clustering.
4. Create cohorts: group remaining failures by type and rank them by frequency × severity.
5. Write one evaluator per cohort: use code checks first, with an LLM judge when code cannot determine the result.
6. Replay after a change: compare against the baseline and add the cases and evaluators to the regression suite.

Replay covers failures already represented in the recordings. ZenML’s regression-suite guide describes synthetic fixtures as complementary coverage for conditions that have not occurred in production.

Model changes are another regression target. Iusztin warned that a model with better benchmark results could reintroduce an old tool-use bug:

ZenML’s replay guide separates a comparison into three sessions:

1. Observed: the original recorded or imported run.
2. Reproduced: an unchanged replay that establishes whether the baseline holds up.
3. Forked: a replay with a changed model, prompt, sampling parameter, or agent version.

The guide identifies live tool traffic and unpinned nondeterminism as possible reasons an unchanged reproduction can diverge from the original.