OpenAI's Pando Cuts Push Latency 80% Serving a Petabyte of Git Daily
OpenAI built Pando, a Git-compatible backend that cut push latency ~80% and now moves nearly a petabyte of Git traffic daily across its monorepo.
- OpenAI unveiled Pando, a Git-compatible backend powering its entire monorepo.
- Median push latency dropped ~80%, system now serves nearly a petabyte of Git data daily.
- Engineers and Codex agents keep using standard Git commands with no workflow changes.
- Parallel branch writes, reusable packfiles, and LSM-tree-style compaction up to L4 covers.
- Fixes are being upstreamed to Git, including interrupted packfile cleanup.
- Built in response to agent-driven commit volume overwhelming conventional Git servers.
OpenAI has detailed Pando, the distributed backend that now serves its main monorepo. Engineers and Codex agents continue using familiar commands such as git push, git fetch, and git clone. Behind that interface, Pando replaces conventional Git server components with services that scale writes, metadata, and object storage horizontally.
OpenAI reports that Pando reduced median push latency by nearly 80 percent, an almost fivefold improvement, and delivers close to one petabyte of Git data per day with a handful of CPU cores. The previous backend had fallen below 90 percent reliability as concurrent agent and human activity produced slow fetches, failed merges, and heavier write contention.
| Layer | Approach | Effect |
|---|---|---|
| Client | Standard Git commands and protocol | Existing workflows remain intact |
| Writes | Parallel updates across separate branches | Independent pushes no longer share one repository-wide queue |
| Reads | Precomputed, reusable packfiles | Fetches avoid repeated pack generation |
| Storage | Separate metadata and blob stores | Capacity grows by adding distributed resources |
| Maintenance | Background packfile compaction | Small writes become larger, read-optimized files |
Git stays on every developer machine
OpenAI traced its reliability problems to the server architecture while retaining Git’s client behavior and wire protocol. That boundary preserves integrations with editors, scripts, continuous integration systems, and coding agents that already understand Git.
The company distributes its own Git build to developer machines and inside the ChatGPT desktop app. Its client can download and index independent packfiles in parallel, accelerating fetches while preserving the commands engineers already use.
Branches stop sharing one lock
A Git branch is represented by a reference, or ref, that points to a commit. Pando allows pushes to different refs to commit concurrently, while updates to the same ref remain serialized so that one branch cannot acquire conflicting histories.
Pando also records refs as a sequence of versions. It maps Git’s “have” and “want” negotiation onto those versions, allowing the service to calculate missing history while many branches advance at once.
Write capacity grows by adding Pando server instances, metadata-store scaling units, or blob storage. This arrangement removes the single-server ceiling that constrained the previous backend.
Packfiles become a shared cache
A Git packfile is a compressed bundle of commits, file contents, and related objects. Conventional servers often build a fresh pack for each fetch, consuming substantial CPU even when many clients request similar histories.
During a Pando fetch, the client reports the commits it has and the ones it wants. The server responds with links to stored packfiles covering the missing history and any required branch dependencies. Many clients can reuse those files, giving the read path the delivery pattern of a content distribution network.
Frequent pushes initially create small packfiles, which background jobs merge into progressively larger “covers.” The process resembles compaction in log-structured merge trees used by databases such as RocksDB and LevelDB: writes land quickly, then asynchronous work reorganizes them for efficient reads. OpenAI reports using covers through level L4.
Git receives the spillover
Protocol compatibility required OpenAI to work directly with Git’s codebase, where the team found client bugs and performance opportunities. Its changes include parallel handling of independent packs and cleanup after an interrupted packfile download.
OpenAI says it is contributing these Git patches upstream. Teams that never deploy Pando could eventually receive some of its client-side improvements through future Git releases.
Why agents exhausted the old backend
OpenAI’s monorepo, a single repository containing most of the company’s code, consists largely of Python, with a growing collection of Rust services and several Go services for systems such as network proxies. Codex agents add many concurrent branches, commits, pushes, and fetches to the activity generated by human engineers.
- Pack generation: Creating compressed object bundles consumed substantial CPU and generally used one thread per request.
- Ref coordination: Updates waited on repository-wide serialization even when pushes targeted separate branches.
- Duplicate work: Fetches repeatedly generated similar packs for different clients.
- Repository size: A large shared history increased the cost of each operation.
Pando addresses those constraints with branch-level write coordination, reusable packs, parallel client downloads, and independently scalable metadata and blob storage.
The deployment gap remains wide
Pando remains an internal system built for OpenAI’s monorepo and infrastructure. The company has released no source code, self-hosted package, public API, or deployment guide. Operating a similar design would also require distributed metadata services, blob storage, compaction jobs, monitoring, and failure recovery.
The performance figures come from OpenAI’s production environment, so they do not establish how Pando would behave with other repository shapes or workloads. GitHub also contains an unrelated project with the same name.
A pattern for agent-heavy repositories
OpenAI is evaluating Pando for additional internal workloads where large numbers of agents collaborate through a shared Git repository. Rising agent activity changes source-control demand by increasing concurrent branch creation, pushes, and repeated fetches far beyond typical human-only patterns.
For infrastructure teams approaching similar limits, Pando provides a concrete architectural pattern: preserve Git’s commands and protocol, coordinate writes at branch granularity, precompute reusable packfiles, and scale metadata separately from object storage. Its immediate external impact will come through upstream Git improvements; its broader design remains a reference until OpenAI publishes enough material for others to implement or test it.
Comments
