Architecture
Three sites, one quorum, a GHES 3.22 front
No Cloudflare storage product promises zero data loss across regions, so Graftyard builds it. Every repository is a small consensus group with one member on each of three continents. A push counts only once two continents hold it.
The path of a push
- The client reaches the nearest Cloudflare data center. The edge authenticates the token and finds the repository's group.
- The pack is checked by upstream git and written to R2 buckets on two sites.
- The leader runs the push through
decide, which checks every ref's expected value and the rulesets, then sends the entry to both followers. - The first follower to store it makes a quorum on two continents. Only then does git hear
ok.
Every repository is a group
A site is one Worker script whose Durable Objects, containers and pack bucket are pinned to one region: North America (enam), Europe (weur) or Asia-Pacific (apac). Members reach each other through Durable Object bindings, not public HTTPS.
Repositories, fork networks, an owner's permissions, token shards and Actions state are each their own group. A group has one entry in flight, and every command that arrives meanwhile joins the next entry.
Issues, pull requests and reviews are commands on the same group as the pushes, so a merge and a review can never disagree about order.
Rules the code is not allowed to break
Each one is enforced in types or checked in tests, and a change that breaks one is wrong even if every other test passes.
- Ack after quorumA success reply can only be built from proof that two sites hold the entry.
- Events are the logAn event exists if and only if its change committed. Webhooks read the log through cursors.
- One entry in flightA follower checks entry k+1 only after it applies k.
- Pure decisions
decidesees only the fold and the commands before it in the same entry. - GHES unchangedNo GHES 3.22 reply changes for a client that does not opt in to the agent layer.
- No API on the hot pathNo request calls the Cloudflare API, so an API outage cannot stop pushes.
When something fails
What a developer notices in normal operation is one intercontinental round trip on each push. Everything else is what happens when parts of Cloudflare fail.
| Failure | What Graftyard does | Effect |
|---|---|---|
| A data center inside a site | Absorbed by Durable Object storage and R2 inside the region | No effect |
| A whole region lost | The other two sites still form a quorum; the group elects a leader there | Measured 11.0 to 12.9 s, then normal |
| A region slow or erroring (gray) | Sites report a sick member; a busy leader steps down on its own | Leaders move off within about 20 s |
| A site deploy | The site is drained first: leaders transfer away before it resets, one site a day | Targets zero failed writes |
| R2 down everywhere | Pushes under 1 MiB ride inside the log entry instead of a pack bucket | Small pushes still land |
| Cloudflare down everywhere | An optional lifeboat on the customer's own hosts serves clones and reads to enrolled developers | Read-only until it returns |
Cloudflare products only
One signed artifact runs managed, in your own Cloudflare account, or self-hosted. Workers KV, D1, Artifacts and K2 are deliberately not used.
| Product | Used for |
|---|---|
| Workers | The front door, the edge (HTTPS git, REST, GraphQL), three site scripts, and graft-ops for signed operations |
| Durable Objects, SQLite-backed | Group members: each holds a Raft log, a lease and the repository's folds (issues, pull requests, reviews) |
| R2 | Pack files in one bucket per site, the clone cache, snapshots, and the vault's locked copy |
| Containers | Upstream git for index-pack, upload-pack, merge-tree and repack, in a git cache per site |
| Queues, Workflows, Cron Triggers | Webhook and search fan-out from the log; member replacement and imports; sweepers |
| Tunnel, DNS, WAF, Secrets Store | Opt-in SSH, the zone, abuse limits, and the keys that sign operations |