Raindrop opens Simulations for agent changes on every pull request

• Raindrop has put Simulations into early access for pre-merge agent tests that create whole environments around an agent, according to benhylak's early-access announcement.
• The system is designed to replay thousands of historical traces against a changed harness, as benhylak's simulation description puts it.
• State fidelity across a changing tool surface is a central constraint, with benhylak's note on harness changes naming the addition of a tool as an example.
• A $50 million total-funding announcement from benhylak accompanies a general-availability target over the coming month, stated in benhylak's availability update.

Raindrop's Simulate page promises existing instrumentation without an SDK rewrite, and lists Langfuse and Helicone as examples. Its sample pull request reports a per-run cost shift from $0.042 to $0.051, or 21.4 percent.

Raindrop says Simulations builds a complete world around an agent for each pull request, including faithful mocks of the services and databases it calls. The Simulate page says the agent's real code runs unchanged against stateful service copies, with tool calls and state changes observable without using production.

Raindrop's existing product collects production-run messages, tool calls, retries, and errors. The company's September 17 funding release says Simulations replays real production traffic and existing test cases against a proposed agent-harness change, then applies its anomaly detection to the results.

That gives a changed agent a stateful run rather than an isolated prompt-and-response check. Raindrop's stated scale target is historical-trace replay in the thousands on a completely new harness.

Raindrop's launch demo reports that its bot ran 46 scenarios against one pull request and flagged three issues:

• Increased sycophancy in a mental-health setting.
• Failures in a new read_pdf tool.
• Skipped tool calls in a design-review scenario.

The technical hinge in benhylak's note on harness changes is preserving the simulated world when the harness changes, including when an engineer adds a tool.

Almost no one was doing meaningful CI evaluations, benhylak said in a reply about CI evaluations.

Raindrop calls the product early access in benhylak's early-access post, while its September 17 release calls it a research preview. General availability is expected over the coming month, according to benhylak's availability update.

The concurrent Series A was led by CRV and brought Raindrop's total funding to $50 million, with Lightspeed also involved.