Durability

When Graftyard says yes to a push, the push exists

No Cloudflare storage product promises zero data loss across regions. So Graftyard built its own replication on Durable Objects, made the guarantee something the compiler checks, and then spent 104.6 hours trying to break it on Cloudflare.

  • 0 of 3,588,424acknowledged writes lost in audited chaos runs, with members reset every 8 to 16 s
  • 3 continentshold every repository: North America, Europe and Asia-Pacific
  • 2 regionsmust store a push before git hears ok

The promise

Zero acknowledged writes lost when any one site fails

A site is a region: North America, Europe or Asia-Pacific. Lose a data center, a region, a continent, or a site's release, and every push, merge, review and token change Graftyard acknowledged is still there.

Each repository is a small Raft group with one member on each continent. A push counts once the leader and one follower on another continent have stored it, and its pack bytes are in R2 buckets on two of those sites.

Checked by the compiler

There is no code path that says yes too early

Every success reply, from git's ok to a REST 201, is rendered from an Acked value. Only the code that counted a quorum on two continents can build one, so a handler that skips the quorum does not compile.

A refusal waits for the same quorum, because it can rest on an earlier command in the same entry. Nothing is answered from an entry that might still be lost.

// crates/raft/src/outcome.rs
/// Built only inside `outcome`, from the acks the leader counted.
/// `report_status` and every 2xx renderer take `&Acked`.
pub struct Acked<T> {
    pub index: Index,
    pub out: T,
    quorum: QuorumProof,   // private: no one else can make one
}

How it holds

Six rules, each one closing a way to lose data

  • 01

    One entry in flight per group

    A follower checks entry k+1 only after it applies k. Every command that arrives meanwhile joins the next entry, so order is never in doubt.

  • 02

    Decisions are pure

    Each command is decided from the repository's state and the commands before it, nothing else. Every member and every replay reaches the same answer.

  • 03

    The log is the only record

    An event exists if and only if its change committed. Webhooks, Actions and search read the log through cursors, so none can fire for a write that did not happen, or miss one that did.

  • 04

    A retry never applies twice

    Every request carries an id. A client that lost its answer resends and gets the outcome that committed, even from a new leader.

  • 05

    No Cloudflare API on the request path

    Pushes and fail-over never call the control plane, so the 41-hour API outage of November 2023 would not stop a write.

  • 06

    Releases cannot outrun their readers

    New data formats switch on only after every site can read them. A follower still on the older release that cannot read an entry deposes the leader that wrote it.

Every way to lose a write

What happens when each part fails

Layered from the data center up to Cloudflare itself. The vault and the lifeboat, the last two rows, arrive with Milestones 5 and 6.

What failsWhat Graftyard doesResult
A data center inside a siteDurable Object storage confirms each write on 3 of 5 followers in different data centers, and R2 replicates in the region0 lost, no effect
A whole region, or a continentThe other two sites still hold every acknowledged entry and form a quorum; a leader is elected there0 lost, 11.0 to 12.9 s measured
A region slow or erroring, not downFollowers report a sick leader from two places, and a leader with slow storage steps down on its own0 lost, leaders move within about 20 s
A leader killed after the quorum, before it answersThe client resends under the same request id and gets the outcome that committed, never a second apply12,000 of 12,000 kept their outcome
A site deploy, or a bad releaseOne site a day, drained first. Followers check every entry, and a follower that cannot read one deposes the leader that wrote itTargets 0 failed writes; signed rollback
One member's storage corruptedTwo other members hold the same log, and each voter rehashes its rows weekly against what it applied to catch silent damage0 lost, two copies remain
A branch deleted or force-pushed by mistakeEvery ref value of the last 35 days is kept as a garbage-collection root, so any past state is restorableRestored on day 30, 0 objects missing
R2 failing everywherePushes up to 1 MiB ride inside the log entry instead of a bucketSmall pushes still land
The account itself compromisedA vault in a separate account, with locked buckets, holds snapshots, log tails and packsRestore to a chosen cutoff (Milestone 5)
Cloudflare down everywhereAn optional lifeboat on the customer's own hosts keeps the log and packs and serves clones and readsRead-only until it returns (Milestone 6)

Designed against real outages

GitHub's incidents, and the class each one cannot happen in

The design was checked against 136 incidents from GitHub's availability reports and Cloudflare's published incidents since 2023. These are the ones about data and failover.

What happened at GitHubIn Graftyard
A 43 s partition promoted a primary across the country; 954 writes on one cluster never replicated, and the outage ran 24 h 11 min (2018)There is no primary to promote. A minority side cannot commit, so there are no divergent writes to reconcile
A partial disk failure that automatic failover could not detect (2023)Failure is detected by the protocol, not health checks. A member whose storage stalls stops acking, and a slow leader loses its lease
A dropped column still referenced, and online schema migrations at peak (2025, 2026)No DDL anywhere. Data shapes change by refold, one site at a time, and an entry a follower cannot read deposes its writer
A 32-bit key overflow blocked new review threads for 3.7 h (2026)64-bit indexes everywhere, ids leased in blocks, an alarm at half the space
A merge queue reverted 2,092 pull requests across 658 repositories (2026)Queue tips only fast-forward, a tree-diff check runs before every merge, and any repository restores to an earlier index
A Redis failover left no writable primary (2026)Leadership moves by the same transfer and election the protocol runs every day, not a separate failover path

From GitHub's post-incident analysis of October 21, 2018 and its monthly availability reports, January 2023 to September 2026. The full list, Cloudflare's incidents included, is in design/DESIGN.md, "Downtime".

Tried on the real platform

Then we tried to break it

Not a simulation of Cloudflare: real Durable Objects on three continents, with clients writing from Atlanta, London and Singapore while a driver reset members and moved leaders. Afterwards every site's stored log was read back and every acknowledged write looked up.

TestLoad and faultsResult
Seven chaos runs on Cloudflare, 104.6 h16,258 member resets, 10,680 leadership transfers, 5% of calls failing0 of 3,588,424 audited acknowledged writes lost; 12,304,207 operations checked for linearizability
Leaders killed between the quorum and the reply12,000 kills on Durable Objects and through the git edge12,000 of 12,000 returned the original outcome; 0 applied twice
Every member of 4 or 8 groups reset inside 60 s100 writes/sEvery group committing again within 1.2 s of its last reset
22% of member calls and storage writes failing9 runs, 300 pushes each92.3% to 98.7% of pushes land within three attempts
Restore window: a branch deleted and another force-pushed on day 1, restored on day 30GC and archiving running, every member truncated past day 0Both refs back on all three members; 397 objects read from R2, 0 missing
Raft simulator, harsh and gray faults40,191,804 steps, 2,000 seeds1,000,856 acknowledged writes, each with valid evidence of its quorum
Planted bugs: code mutants run one at a time against the test suites85 in Milestones 0 and 184 of 85 caught; the one missed was a resumed leader sending entries before they were durable, now covered

Every figure is from tests/RESULTS.md, which lists each run with its faults, checks and source. One chaos run found a real bug, a reused ticket after a leader reset; it was fixed and the 48-hour run followed.

Every chaos run