Raindrop launches pull-request agent simulations in sandboxes

• Raindrop brings production-scenario agent tests to every PR, handling sandboxing and fixtures, according to the launch thread.
• Agents run in isolated sandboxes, with connected services recreated using real and synthetic data.
• Simulations also monitor whether agents recognize that they are being tested, according to the announcement.

Raindrop is taking on the ugly part of agent evals: stateful copies of external services, including generated responses for newly added tools with no production history. A separate harness playground runs the full agent harness without opening a PR.

Raindrop pitches one-click tests directly from GitHub. By default, PRs that change prompts, tools, or model settings trigger simulations automatically; commenting @raindrop simulate runs or reruns them, according to the November 5 public-beta announcement.

The GitHub report compares the updated agent with historical production outcomes, including:

• How many replayed conversations exercised the change, and how many improved or worsened.
• Differences in responses, tool calls, and execution paths.
• Tool failures, execution duration, latency, and cost.

A production trace records one execution path. An updated agent can choose another tool or branch, so Raindrop reconstructs the surrounding environment rather than relying on the original sequence of responses, its technical description explains.

The environment has four parts:

1. Fixtures and boundaries. Raindrop handles sandboxing and fixtures, reading the agent’s code to identify its tools and external services.
2. The real agent. Code from the PR runs with its actual prompts, models, and agent loop, without access to production services or credentials.
3. Stateful service copies. Production traces populate simulated dependencies with real and synthetic data. Reads and writes stay inside those copies, including messages, refunds, and database updates.
4. Previously unseen tools. When a PR introduces a tool absent from historical traces, Raindrop generates responses for it. The simulated environment supports paths that differ from the original execution.

Raindrop currently uses Modal, according to a launch reply. Custom evals and Raindrop Signals can still grade each run alongside the production comparison.

Raindrop CTO Ben Hylak says the service monitors “simulation awareness” in real time: whether the agent recognizes that it is under test.

Hylak pointed to OpenAI’s deployment simulations as precedent, while describing harness changes as harder to simulate than model changes. In its June research report, OpenAI describes regenerating assistant responses from previous production conversations with candidate models, extending the method to tool-using agent rollouts.

Production remains the most accurate and faithful place to discover issues and understand a change’s impact, Hylak wrote in the launch thread.

The setup FAQ lists these requirements:

• A GitHub repository containing a TypeScript, JavaScript, or Python agent.
• Agent events already being sent to Raindrop.
• Installation of the Raindrop GitHub app, with read access to the repository and access to the traces.

Raindrop runs the agent’s own code, supporting frameworks including Vercel AI SDK, OpenAI Agents SDK, Mastra, LangChain, and custom loops. Pricing will be set before general availability.

Raindrop reports more than 33,000 simulations across more than 1,700 pull requests since early access began in September. Its customer examples include two specific fixes:

• Caddy: The team tested prompt changes for sarcasm recognition against production conversations. After three or four iterations, Raindrop says the agent recognized jokes without misclassifying ordinary messages.
• LEAP CRM: A proposed fix still left the agent explaining CRM statuses and stages unclearly. The team revised the change, simulated it again, and shipped the improvement.