Evidence
What we tested, and what it showed
Graftyard ran on Cloudflare in test installs across North America, Europe and Asia-Pacific. Real clients hit it, a chaos driver attacked it, and checkers read back every site's log. These are the results through 5 November 2026, failures included.
- 348 of 348git client checks on three git versions, a site failing in half
- 0 of 3,588,424acknowledged writes lost in audited chaos runs
- 104.6 hof chaos on Cloudflare, 12,304,207 operations checked
- 172 to 370 msmedian commit across two continents
- 84 of 85planted bugs caught by the test suites
- 259Rust tests in the workspace, 0 failing
GHES 3.22 compatibility
Unmodified clients, checked against the GHES schemas
No client was patched and no request went to github.com. The host answers /api/v3/meta with installed_version 3.22.0, so gh, git-lfs, VS Code and JetBrains all take their GHES path.
Clients
| Client and commands | Where | Result |
|---|---|---|
gh 2.102.1: issue list / create / view / comment | Hosted edge on Cloudflare, 2026-11-05 | Pass, 5 of 5 exit 0 |
gh 2.102.1: pr list / create --draft / view | Hosted edge | Pass, 3 of 3 exit 0 |
gh 2.102.1: release create / upload / view / download, --archive=tar.gz | Hosted edge | Pass, the asset round-trips byte for byte |
| git-lfs 3.8.0: push, clone, pull, lock, locks, unlock | Hosted edge | Pass, pulled bytes equal pushed bytes |
clients.sh, 29 checks per client | git 2.39.5, 2.51.0, 2.56.0; 12 runs | 348 of 348 rows, with and without a site failing |
git's own smart-HTTP tests, t/t55xx to t/t58xx | git 2.51.0 and 2.56.0, site 0 failing | 0 baseline passes lost on 2.51.0; 1 on 2.56.0 (t5549 #2, a server-side receive.hiderefs) |
Schemas
| Check | Scope | Result |
|---|---|---|
| Live replies on Cloudflare | 26 GraphQL, 12 REST, 5 LFS | 0 failures |
| Replayed client requests | 65 requests from git 2.56.0, git-lfs 3.8.0 and gh 2.102.1; 93 replies | 0 failures |
| Agent layer off by default | 93 replies × 13 Accept values, compared byte for byte with and without the layer | 0 of 1,209 differ |
The GraphQL schema is generated from the GHES 3.22 SDL (1,626 types). gh pr checks exits 1 on a pull request with no checks, which is gh's own answer for that case; every request in it returned 200. VS Code, JetBrains, Terraform, Jenkins and Argo CD are covered by reading their source for the GHES paths they call, not yet by a live run.
Chaos and linearizability
Seven runs with faults on, 104.6 hours
Every 8 to 16 s the driver reset a member or moved leadership. 5% of calls between sites failed and 5% of replication replies were dropped. Clients in Atlanta, London and Singapore kept writing.
- Porcupine checked each history for linearizability, key by key.
- Lost-write audit: after a run, all three sites' stored logs are read back and every acknowledged write is looked up.
The runs on Cloudflare
| Run | Length | Operations | Faults injected | Result |
|---|---|---|---|---|
cf-48h | 48 h | 5,742,736 | 5,729 resets, 3,607 transfers | Pass. 0 of 1,814,873 acked writes lost |
lin24b | 24 h | 2,850,523 | 4,176 resets, 2,752 transfers | Pass. 0 of 904,294 acked writes lost; ack check 0 violations |
lin24 | 24 h | 2,739,691 | 5,547 resets, 3,743 transfers | Not passed. 0 of 869,257 lost, but an early ack was not ruled out; rerun as lin24b |
cf-long-2 | 4 h | 475,301 | 138 resets, 77 transfers | Illegal. A real bug: a reused ticket after a leader reset. Fixed, then cf-48h |
cf-long-1 | 2 h | 270,040 | 243 resets, 186 transfers | Ok |
wp4-cf-5 | 2 h 5 min | 184,349 | 343 resets, 253 transfers | Ok |
wp4-cf-1 | 31 min | 41,567 | 82 resets, 62 transfers | Ok |
Recorded-clock checks flag some cross-site reads as Illegal; with 100 ms of clock allowance, or with the measured bound, every passing run is Ok on all keys. lin24b's 20 flagged segments trace to 15 lost client log chunks and 2 ties needing 1 and 4 ms of skew, against 63 to 92 ms measured that minute.
Targeted fault gates
| Fault | Runs | Result |
|---|---|---|
| Leader killed after quorum, before the reply (BF-12) | 12,000 kills landed | 12,000 of 12,000 kept their original outcome; 0 applied twice |
| Receipts that miss their deadline (BF-14) | 2,400 commands | 2,400 of 2,400 decided once; 0 forwarded |
| One site's member class failing (B0-03) | 26 runs plus 6 reruns | 11.0 to 12.9 s to recover; 0 failed operations |
| 22% of member calls and storage writes failing (B0-25) | 9 runs, 300 pushes each | 92.3% to 98.7% of pushes land within three attempts |
| Every member of 4 or 8 groups reset inside 60 s (B0-26) | 4 runs at 100 writes/s | Pass. Every group commits within 1.2 s of its last reset |
| Simulator, HARSH and GRAY faults | 40,191,804 steps, 2,000 seeds | 1,000,856 acked writes, every one with valid ack evidence |
Latency
What two continents cost a write
A write waits for one round to the nearest site in another region. That is the price of losing nothing when a region goes down. Reads from the leader's region pay no such round.
| Measure | Median | Tail | Gate |
|---|---|---|---|
| One cross-region round, enam to weur, under 100 commands/s | p50 107 to 189 ms | p99 283 to 473 ms | B0-01, Milestone 0 |
| Idle ping, leader to the weur member | p50 135 ms | p99 151 to 216 ms | B0-01, Milestone 0 |
| Commit, one entry in flight, 15 runs | p50 172 to 370 ms | p99 399 to 690 ms | B0-01, Milestone 0 |
| Learner reads, 16 learners at 1,000 reads/s each | 16,000 reads/s, 0 failed | p99 683 to 743 ms | B0-18 |
| CI clones from the R2 clone cache, 33 a second | 100% from R2 with the push hook | time to first byte p99 406 ms | B0-23 |
Clients during the 24-hour chaos run lin24b | p50 314 ms | p99 952 ms | BM-1 |
The commit figures are from Milestone 0's prototype. The production Rust host measured p99 414 to 807 ms in its second pass: 2 of 6 runs under the 600 ms line, none over 1 s. Closing that gap is open work.
Still open
- SSH git sessions held at 99.59% to 99.89% through container kills, between the 99% and 99.9% lines. SSH is opt-in, and its gate is carried to Milestone 6.
- One planted bug of 85 went uncaught: a resumed leader sending entries before they were durable.
- Milestone 2 (pull requests, merge queues, folds) is under way; its gate has not run yet.
Sources: tests/RESULTS.md and the results files it cites, in the Graftyard repository.