Durability
When Graftyard says yes to a push, the push exists
No Cloudflare storage product promises zero data loss across regions. So Graftyard built its own replication on Durable Objects, made the guarantee something the compiler checks, and then spent 104.6 hours trying to break it on Cloudflare.
- 0 of 3,588,424acknowledged writes lost in audited chaos runs, with members reset every 8 to 16 s
- 3 continentshold every repository: North America, Europe and Asia-Pacific
- 2 regionsmust store a push before git hears
ok
The promise
Zero acknowledged writes lost when any one site fails
A site is a region: North America, Europe or Asia-Pacific. Lose a data center, a region, a continent, or a site's release, and every push, merge, review and token change Graftyard acknowledged is still there.
Each repository is a small Raft group with one member on each continent. A push counts once the leader and one follower on another continent have stored it, and its pack bytes are in R2 buckets on two of those sites.
Checked by the compiler
There is no code path that says yes too early
Every success reply, from git's ok to a REST 201, is rendered from an Acked value. Only the code that counted a quorum on two continents can build one, so a handler that skips the quorum does not compile.
A refusal waits for the same quorum, because it can rest on an earlier command in the same entry. Nothing is answered from an entry that might still be lost.
// crates/raft/src/outcome.rs /// Built only inside `outcome`, from the acks the leader counted. /// `report_status` and every 2xx renderer take `&Acked`. pub struct Acked<T> { pub index: Index, pub out: T, quorum: QuorumProof, // private: no one else can make one }
How it holds
Six rules, each one closing a way to lose data
- 01
One entry in flight per group
A follower checks entry k+1 only after it applies k. Every command that arrives meanwhile joins the next entry, so order is never in doubt.
- 02
Decisions are pure
Each command is decided from the repository's state and the commands before it, nothing else. Every member and every replay reaches the same answer.
- 03
The log is the only record
An event exists if and only if its change committed. Webhooks, Actions and search read the log through cursors, so none can fire for a write that did not happen, or miss one that did.
- 04
A retry never applies twice
Every request carries an id. A client that lost its answer resends and gets the outcome that committed, even from a new leader.
- 05
No Cloudflare API on the request path
Pushes and fail-over never call the control plane, so the 41-hour API outage of November 2023 would not stop a write.
- 06
Releases cannot outrun their readers
New data formats switch on only after every site can read them. A follower still on the older release that cannot read an entry deposes the leader that wrote it.
Every way to lose a write
What happens when each part fails
Layered from the data center up to Cloudflare itself. The vault and the lifeboat, the last two rows, arrive with Milestones 5 and 6.
| What fails | What Graftyard does | Result |
|---|---|---|
| A data center inside a site | Durable Object storage confirms each write on 3 of 5 followers in different data centers, and R2 replicates in the region | 0 lost, no effect |
| A whole region, or a continent | The other two sites still hold every acknowledged entry and form a quorum; a leader is elected there | 0 lost, 11.0 to 12.9 s measured |
| A region slow or erroring, not down | Followers report a sick leader from two places, and a leader with slow storage steps down on its own | 0 lost, leaders move within about 20 s |
| A leader killed after the quorum, before it answers | The client resends under the same request id and gets the outcome that committed, never a second apply | 12,000 of 12,000 kept their outcome |
| A site deploy, or a bad release | One site a day, drained first. Followers check every entry, and a follower that cannot read one deposes the leader that wrote it | Targets 0 failed writes; signed rollback |
| One member's storage corrupted | Two other members hold the same log, and each voter rehashes its rows weekly against what it applied to catch silent damage | 0 lost, two copies remain |
| A branch deleted or force-pushed by mistake | Every ref value of the last 35 days is kept as a garbage-collection root, so any past state is restorable | Restored on day 30, 0 objects missing |
| R2 failing everywhere | Pushes up to 1 MiB ride inside the log entry instead of a bucket | Small pushes still land |
| The account itself compromised | A vault in a separate account, with locked buckets, holds snapshots, log tails and packs | Restore to a chosen cutoff (Milestone 5) |
| Cloudflare down everywhere | An optional lifeboat on the customer's own hosts keeps the log and packs and serves clones and reads | Read-only until it returns (Milestone 6) |
Designed against real outages
GitHub's incidents, and the class each one cannot happen in
The design was checked against 136 incidents from GitHub's availability reports and Cloudflare's published incidents since 2023. These are the ones about data and failover.
| What happened at GitHub | In Graftyard |
|---|---|
| A 43 s partition promoted a primary across the country; 954 writes on one cluster never replicated, and the outage ran 24 h 11 min (2018) | There is no primary to promote. A minority side cannot commit, so there are no divergent writes to reconcile |
| A partial disk failure that automatic failover could not detect (2023) | Failure is detected by the protocol, not health checks. A member whose storage stalls stops acking, and a slow leader loses its lease |
| A dropped column still referenced, and online schema migrations at peak (2025, 2026) | No DDL anywhere. Data shapes change by refold, one site at a time, and an entry a follower cannot read deposes its writer |
| A 32-bit key overflow blocked new review threads for 3.7 h (2026) | 64-bit indexes everywhere, ids leased in blocks, an alarm at half the space |
| A merge queue reverted 2,092 pull requests across 658 repositories (2026) | Queue tips only fast-forward, a tree-diff check runs before every merge, and any repository restores to an earlier index |
| A Redis failover left no writable primary (2026) | Leadership moves by the same transfer and election the protocol runs every day, not a separate failover path |
From GitHub's post-incident analysis of October 21, 2018 and its monthly availability reports, January 2023 to September 2026. The full list, Cloudflare's incidents included, is in design/DESIGN.md, "Downtime".
Tried on the real platform
Then we tried to break it
Not a simulation of Cloudflare: real Durable Objects on three continents, with clients writing from Atlanta, London and Singapore while a driver reset members and moved leaders. Afterwards every site's stored log was read back and every acknowledged write looked up.
| Test | Load and faults | Result |
|---|---|---|
| Seven chaos runs on Cloudflare, 104.6 h | 16,258 member resets, 10,680 leadership transfers, 5% of calls failing | 0 of 3,588,424 audited acknowledged writes lost; 12,304,207 operations checked for linearizability |
| Leaders killed between the quorum and the reply | 12,000 kills on Durable Objects and through the git edge | 12,000 of 12,000 returned the original outcome; 0 applied twice |
| Every member of 4 or 8 groups reset inside 60 s | 100 writes/s | Every group committing again within 1.2 s of its last reset |
| 22% of member calls and storage writes failing | 9 runs, 300 pushes each | 92.3% to 98.7% of pushes land within three attempts |
| Restore window: a branch deleted and another force-pushed on day 1, restored on day 30 | GC and archiving running, every member truncated past day 0 | Both refs back on all three members; 397 objects read from R2, 0 missing |
| Raft simulator, harsh and gray faults | 40,191,804 steps, 2,000 seeds | 1,000,856 acknowledged writes, each with valid evidence of its quorum |
| Planted bugs: code mutants run one at a time against the test suites | 85 in Milestones 0 and 1 | 84 of 85 caught; the one missed was a resumed leader sending entries before they were durable, now covered |
Every figure is from tests/RESULTS.md, which lists each run with its faults, checks and source. One chaos run found a real bug, a reused ticket after a leader reset; it was fixed and the 48-hour run followed.