Pi-durable reports 439 runs per second after batching database commits

• Batching database commits raised single-session throughput from 4.8 to 439 synthetic runs/s at 128 concurrent conversations in badlogicgames' benchmark.
• Median time inside the harness fell from roughly 25 seconds to 0.28 seconds at that load, according to the latency results.
• Local SQLite improved 18–58%; Cloudflare Durable Objects read 40% fewer rows and wrote 13% fewer in the backend tests.
• The tests used instant fake models and no-op tools, as the developer emphasized. Real model calls and tool execution add their own costs.

Normal runs now avoid database reads altogether, alongside the commit batching. The clever bit is splitting a batch when a later commit depends on data an earlier one changes. Pi-durable still carries an experimental API warning.

Pi-durable stores agent progress before execution advances, allowing interrupted work to resume from committed checkpoints. The illustrated run requires about 12 commits across input admission, model calls, tool execution, and completion.

One session can host many conversations, including subagents or separate Slack threads. Previously, it sent every commit to storage individually and waited for the response, leaving conversations queued behind one another.

The throughput test used 4 ms storage round trips, described as comparable to managed Postgres latency. Its numbers measure total throughput across the session, not throughput per conversation.

The batcher follows three rules:

1. Commits arriving while a storage call is in flight accumulate for the next call.
2. It tracks what each commit mutates using ad-hoc dependency tracking.
3. A later commit that reads or mutates the same data forces a split, as the developer explained.

At 128 conversations, a storage call carries about 33 commits. Each retains its own sequence number and success or failure; unpersisted changes remain invisible.

Two accompanying changes reduce storage work further:

• A record cache eliminates database reads during a normal run.
• The example run needs 12 commits instead of 14.

Parallel conversations can inherit earlier history through forks. The developer described the underlying structure in a reply:

• Transcript entries, including user messages, assistant messages, tool results, and custom entries, are immutable and ordered.
• Each conversation is an ordered list of those entries.
• A new conversation can fork from an entry in another conversation, inheriting the shared prefix.

The gains extended to local storage and Cloudflare's SQLite-backed Durable Objects in the same test series.

Local SQLite, agent runs/s on one CPU core

• One conversation: 120 → 190, +58%.
• 32 conversations: 284 → 334, +18%.
• 128 conversations: 306 → 385, +26%.

Cloudflare Durable Objects, storage work per run

• Rows read: 241 → 145, −40%.
• Rows written: 158 → 137, −13%.
• SQL statements: 188 → 97, −49%.

Cloudflare also bills active compute duration separately from storage, so those row reductions describe only part of an object's resource usage.

Following an exchange with @she_llac, the developer followed the storage work with a CPU optimization. After every commit, the scheduler had been rechecking every conversation active within the previous ten minutes.

The replacement checks only conversations touched by the commit, with separate small indexes for sleeping tasks and expiries.

The follow-up reported:

• 4,096 conversations, in-memory storage: 46 → 772 runs/s.
• CPU time per run at that load: about 22 → 1.6 ms.
• 2,048 conversations, local SQLite: 88 → 459 runs/s.
• 2,048 conversations, three parallel no-op tools: 58 → 617 runs/s.
• One conversation: unchanged.

These tests used one CPU core and in-memory storage unless otherwise noted. Every conversation had run once before measurement, so the figures describe warmed harness performance.

After the storage pass, the developer had identified single-core CPU saturation as the remaining limit on scaling within one Node process.

The storage pass left resting memory within ±2 MB at 128 conversations, with caches capped by count or live work, according to the accompanying chart.

With local SQLite and 4,096 conversations, the CPU follow-up reported:

• Resting heap: 100 → 100 MB.
• Resting RSS: 902 → 851 MB.
• Peak heap during runs: 648 → 604 MB.
• Throughput: 47 → 444 runs/s.

The changelog also documents documentRetentionMs: loaded documents without observers leave memory after ten minutes without reads or writes by default; setting it to 0 drops them after use.