OpenAI says Pando cuts monorepo Git push latency by 80%

• Pando now powers OpenAI’s monorepo with unchanged Git workflows and roughly 80% lower median push latency, according to OpenAI’s announcement.
• Independent branch updates and reusable packfiles supply the scale, with OpenAI’s thread spelling out two tradeoffs: mixed-time branch tips and some redundant downloads.
• External access is under consideration, according to gdb’s follow-up, rather than an announced public release.

OpenAI’s cache figures are wonderfully lopsided: more than 57 TB served in one hour, with over 99% of bytes coming from local disk cache. A Git patch series traces a pack-corruption bug to two processes appending bytes to the same temporary download file.

Codex multiplied concurrent changes, and each push could trigger several CI jobs fetching the same commit. In OpenAI’s engineering report, the previous monorepo service suffered slow pushes and fetches, with merge reliability sometimes falling below 90%.

The report also compares Pando with three-phase commit and a blob-store write-ahead log using an interactive simulation, including a configuration with 1,000 writers on distinct branches. That comparison is explicitly labeled a synthetic teaching model, separate from the production latency claim.

Pando separates immutable pack storage from branch metadata. A push follows three steps in OpenAI’s architecture description:

1. Validate the incoming pack while streaming it to blob storage, without unpacking it into a local repository.
2. Record a ref version containing the new tip, its pack and dependencies on earlier ref versions.
3. Atomically update the ref pointer in a metadata transaction.

OpenAI reports constant-time metadata reads and writes per push, plus read-your-own-write consistency.

Updates to unrelated branches proceed independently; updates to one ref remain serializable. As OpenAI explains, a fetch spanning several branches can observe their tips at different points in time, without a repository-wide snapshot.

Fetch negotiation and bulk delivery take separate paths:

1. The client identifies the commits it has and wants.
2. Pando selects stored packs covering the missing history, including dependencies from other branches.
3. The server returns packfile URIs; the client downloads through Express, a service with regional caches.

The client may receive objects it already has. Reusing packs avoids generating a tailored pack for every reader, while Pando servers avoid applying incoming packs to local repository replicas.

OpenAI says the service delivers nearly a petabyte daily, with only a handful of CPU cores needed to sustain that delivery rate in its production account.

Pando uses hierarchical “covers” to describe groups of stored packs, while background compaction combines packs into larger reusable units. Its fetch-planning design avoids walking every historical push individually.

OpenAI reports O(log Δ) metadata reads per fetch, where Δ is the shortest distance in ref versions between the client’s known history and its target. Downloading, verifying and indexing the selected objects remain separate work, a distinction highlighted in an independent technical walkthrough.

The Rust backend, built with Codex, reached production after ten weeks of development and began serving traffic in April, according to the walkthrough of OpenAI’s report. Validation included conformance tests, deterministic simulations, formal specifications and agent red-teaming.

The July 26 patch series also addresses a double-close involving a staging-file descriptor passed to index-pack. Another fix handles a resumed download whose temporary file already contains the complete pack, causing the repeat HTTP Range request to return 416.

OpenAI separately says it contributed fixes for cleanup after interrupted packfile downloads.