Plurai Simulation replays failing agent turns with restored sandbox state

• Failing agent turns can be replayed with conversation history and sandbox records restored, as the replay description explains.
• Refund evaluations check simulated billing records alongside the agent's response, according to the billing-agent example.
• Plurai Simulation is live and free to start, according to launch coverage.

Plurai's replay account includes an agent authorized to refund $200 offering a customer $550. Its architecture post describes a tool-routing mode that executes real calls. And despite early coverage listing Cursor, the installation docs say Cursor support is coming soon.

Replay imports a production trace containing messages, tool calls and tool results. The clever boundary is around the model: Plurai freezes the conversation and surrounding world while generating the response again.

The replay write-up describes three pieces:

• History: earlier turns are supplied exactly as recorded, without regenerating them.
• World state: mocked tools and their backing records are reconstructed to the moment immediately before failure.
• Failing turn: the agent generates it again, normally across three trials to distinguish consistent failures from intermittent ones.

The model remains nondeterministic. Prompt edits, model swaps and tool changes can face the same frozen starting point, and the replay can become a regression scenario.

In Plurai's reported billing-agent case, six trials repeated the prohibited refund offer. Two continuations reached an actual refund-tool call before the tool's limit blocked it, exposing behavior beyond where the original production trace ended.

A refund promise can sound successful while leaving the billing system unchanged, the failure illustrated in the billing-agent example. Plurai gives each scenario shared state across its mocked tools.

Its architecture requires three things to agree:

• Customer story: an order mentioned by the simulated user exists with the charges they describe.
• Tool state: a refund_charge write becomes visible to the next get_order call.
• Expected outcome: the evaluator checks the resulting records against what should have happened.

The architecture post's example goes beyond detecting a missing call: refunding both charges instead of only the duplicate would fail, with the two refunded rows included as evidence.

Simulation, grading and a proposed refund-agent fix

Plurai launched on November 11. Its documented plugin surfaces are Claude Code, Codex CLI and GitHub Copilot CLI, with a Python SDK underneath.

The coding agent handles the six-step loop described in a walkthrough made with the team:

1. Read the agent: scan code, prompts, tools and policies with /simulation-scan-agent.
2. Map the rules: build a world graph of entities, tool reads and writes, flows and constraints. Conflicting sources are flagged, with code taking precedence over traces, then the generated PRD.
3. Generate scenarios: /simulation-generate-suite builds the mock world, user scenarios and evaluation criteria together.
4. Run conversations: /simulation-run starts a separate agent process and mock session per scenario, with scenarios running in parallel.
5. Grade behavior: code checks inspect final world state; LLM judges handle criteria requiring judgment.
6. Propose fixes: /simulation-fixes identifies failures and proposes file-and-line changes. Code changes require acceptance.

The scan creates plurai_sim/ beside the agent, including configuration, a plain-language PRD, scenario variables and a generated simulate.py harness. The docs also expose /simulation-fixes --verify to rerun the suite and mark fixes as resolved or still failing.

A tool combining reads and writes prompted a question about how the scan chooses which calls stay live. Plurai's architecture post documents three modes, selected per external system:

• mock: route calls to the simulated world.
• real: execute the agent's own tool code against scenario-seeded data.
• observe: execute the real tool and report its result to the simulated world.

The scan proposes boundaries for approval. Plurai gives mocked payments and email alongside a live read-only knowledge base as an example; the post does not detail separate routing for reads and writes inside one mixed-purpose tool.

The agent runs locally, but the simulation has a hosted backend. Plurai splits the architecture into three components:

• Plugin: reads the repository and writes the harness through the coding agent; communicates with the backend over MCP.
• SDK: starts the local agent and drives turns; communicates over REST.
• Backend: builds the world graph, mock environment, scenarios and evaluations, then hosts the simulation services.

According to Plurai's architecture post, model keys and tool credentials remain on the local machine. The backend receives the plurai_sim/ folder, optional sources selected for inclusion and simulated conversations during runs.

Knowledge-base coverage also depends on those optional inputs: without a supplied knowledge source, Plurai says the knowledge base goes untested.