Evidence / All results
Every check, benchmark and baseline
This page is the repository's own record, unedited: the results ledger, then each milestone's status and the pass and fail lines of every work package's gate. It lists 380 rows in 96 tables, failures and unrun rows included. For the short version, see the overview.
- Results ledger, Milestones 0 and 1
- Milestone 0: measure the platform and the protocol
- Milestone 1: groups and git
- Milestone 2: folds, issues, pull requests, and merge
- Milestone 3: identity and authorization
- AM1: base agent changes, leases, and tasks (built with M2 and M3)
- Gates not yet run (7 milestones)
Built 2026-11-15 from tests/RESULTS.md and build/BUILD-PLAN.md. Paths in the tables are files in the Graftyard repository. Evidence tags: measured was run; doc comes from vendor documentation; inference and guess are reasoned, not run.
tests/RESULTS.md
Results ledger, Milestones 0 and 1
This file lists every run that counts toward a gate row or a decision in Milestones 0 and 1. It covers runs on Cloudflare, in the Raft simulator, and through the Oracle relay. It gives each run's gate row, place, dates, load, faults, checks, and result, and cites the results file and section the numbers come from. Smoke runs and discarded runs appear only in "Smoke, local, and discarded runs", labelled as such. Every figure is measured unless a row says otherwise.
Raw data paths (raw/) are git-ignored and exist only on the machine that ran the test. Results files cite them. Where this file and a results file's prose disagree, this file takes the raw value, and "Prose and raw data that disagree" lists the difference.
Totals
Each total has one definition. Totals add only counts of the same thing from different runs, never the counts of different checkers on one history.
| Total | Value | Definition |
|---|---|---|
| Operations checked for linearizability on Cloudflare | 12,304,207 | Operations in the histories of the seven chaos runs in "Linearizability and chaos runs on Cloudflare". Each history counts once, though most were checked two or three ways (recorded clocks, widened 100 ms, widened by the measured bound). By verdict: 9,089,215 Ok (cf-long-1, cf-48h widened 100 ms, wp4-cf-1, wp4-cf-5, lin24b). 475,301 Illegal from a real bug (cf-long-2). 2,739,691 not passed (lin24: Illegal on the recorded clocks, Ok widened, early ack not ruled out). Excludes the smoke runs, BF-13's 3,680 operations, and every simulator history |
| Hours of chaos on Cloudflare | 104.6 h | Wall-clock length of those seven runs: 2 + 4 + 48 + 0.52 + 2.08 + 24 + 24. The windows include stretches without chaos: four of 68 to 115 min in cf-48h, three driver gaps in cf-long-2, a fault plan overwritten for part of cf-long-1, and call faults on only 39 of 125 min in wp4-cf-5 |
| Member resets injected | 16,258 | /reset steps by the chaos driver in those seven runs: 243 + 138 + 5,729 + 82 + 343 + 5,547 + 4,176. cf-long-2's 138 comes from prose, because its event log was overwritten |
| Leadership transfer attempts injected | 10,680 | Transfer steps the chaos driver sent in those seven runs: 186 + 77 + 3,607 + 62 + 253 + 3,743 + 2,752, including planned abandons and attempts answered "not leader" or refused by a quarantined target. Transfers that moved the leader are not totalled, because M0's driver and M1's report classify outcomes differently (see each row) |
| Acknowledged writes audited against the members' logs | 3,588,424, 0 lost | cf-48h 1,814,873, lin24 869,257, and lin24b 904,294. Each audit reads all three members' stored logs after the run |
| Leader kills after quorum and before the reply | 12,000 landed | BF-12 on Durable Objects: 10,000 (bf12-full), 1,000 (bf12-p3), and 1,000 on Push through the git edge (p2). Every one returned its original outcome |
| SSH git sessions | 31,482, 31,362 succeeded | B0-24 (2,637), B0-24a r4 (28,347), and the Oracle relay run r1 (249 through the relay plus 249 control sessions through cloudflared). Excludes three stopped 24-hour attempts (1,144 sessions) |
Results of git's own t/ tests on Cloudflare | 2,667 | Rows in results.tsv of the no-fault baseline (1,067), the three BM-1 faulted runs (1,558), and the t5541 recheck (42), skips included. Per version below |
clients.sh runs | 12 runs, 348 of 348 rows | 29 rows each, on git 2.39.5, 2.51.0, and 2.56.0, with and without a site failing |
| Tests in the workspace | 259 Rust, 14 Go | cargo test --workspace at c0c6fb9: 259 passed, 0 failed, 2 ignored (long_sweep, long_liveness). Go: tests/m0/porcupine 11, containers/git/server 3 |
| Simulator long sweep | 40,191,804 steps, 1,000,856 acked writes | One run of long_sweep at 2,000 seeds × 10,000 steps, HARSH plus GRAY. Every acked write passed the in-simulator ack check. Earlier sweeps are listed, not added |
| Planted bugs caught | 84 of 85 | Code mutants planted one at a time against a test suite. Planted bad histories fed to a checker are counted apart |
Linearizability and chaos runs on Cloudflare
Every run used the same fault plan unless its row says otherwise. 5% of member-to-member calls fail from each site, and 5% of AppendEntries replies are dropped. A chaos loop resets a member or transfers leadership every 8 to 16 s, and some transfers are abandoned on purpose. Load clients are pinned in ATL, LHR, and SIN unless the row names the orb (LAX). Porcupine is tests/m0/porcupine, run per key with segmented mode (-seg N).
scripts/results_lin.py <run>.report.json prints the lin24 row's first seven columns from the run's report and event log. Use it for every later lin run.
| Run | Gate | Where and when | Duration | Operations | Faults | Checks | Result | Source |
|---|---|---|---|---|---|---|---|---|
cf-long-1 | B0-11 | Cloudflare, graft-raft-*, 2026-10-28T08:42Z to 10:42Z | 2 h | 270,040, 0 failed; p50 218 ms, p99 1,033 ms | 243 resets. 186 transfer attempts: 81 transferred, 46 "not leader", 5 abandoned with the reply lost, and 54 planned abandons (17 of them answered "not leader"). The B0-06 and B0-03 runs overwrote the fault plan for parts of 09:34Z to 10:25Z, and one deploy reset every member at 09:33Z (raft-chaos-cf-long-1.started) | Porcupine on the whole history, rechecked with unique write names | Ok | tests/m0/results/m0-wp2.md, "B0-11: the 4-hour run found a linearizability violation"; raw/raft-chaos-cf-long-1.json |
cf-long-2 | B0-11 | Cloudflare, graft-raft-*, 2026-10-28T13:00Z to 17:00Z | 4 h | 475,301 | 138 resets and 77 transfers (prose only). No resets or transfers in three driver gaps, and one deploy reset every member at 14:44Z (raft-chaos-cf-long-2.started) | Porcupine; members' stored logs read back with /dump; deterministic repro (run/raft-ticket.ts, 3 of 3) | Illegal. A real bug: a Ticket of (term, seq) was reused after a leader reset, so outcome answered an old ticket with another command's result. Fixed in the design and the prototype by adding boot | tests/m0/results/m0-wp2.md, "B0-11: the 4-hour run found a linearizability violation"; raw/raft-chaos-cf-long-2.json |
cf-48h | B0-11 | Cloudflare, graft-raft-*, 2026-10-28T18:50Z to 2026-10-30T18:50Z | 48 h | 5,742,736, 12 with no reply; p50 284 ms, p99 1,211 ms | 5,729 resets (A 2,323, B 2,292, C 1,114). 3,607 transfer attempts, 1,265 "not leader", so 2,342 from a leader. Of those, 1,431 transferred, 779 abandoned, and 132 transferred with the reply lost. No per-isolate counters. Coverage on site C is thin (F36) | Porcupine -seg 20000 per key. Lost-write audit of all three logs. Single-clock early-ack test | Recorded clocks Illegal on all 6 keys, from 72 cross-colo reads 27 to 94 ms stale, all consistent with clock skew. Widened 100 ms Ok on all 6 keys. 0 of 1,814,873 acked writes lost; 984,514 identical entries on 3 members. Early-ack test 0 of 72 and 0 of 5,934 controls. Pass with the 100 ms allowance as an assumption, not a measured bound. Rechecked with the fixed segmented checker: unchanged | tests/m0/results/m0-wp2.md, "B0-11: the 48-hour run on the fixed Ticket code"; tests/m1/results/porcupine-seg.md; raw/raft-chaos-cf-48h.events.jsonl, raw/cf-48h-analysis/lost-acked.json |
wp4-cf-1 | B0-11 (fault and transfer rules on the M0-WP4 code) | Cloudflare, graft-wp4-*, 2026-10-30T00:09Z to 00:40Z, 8 clients from the orb | 31 min | 41,567 (4 failed at the orb's own fetch); p50 287 ms, p99 1,090 ms | 82 resets. 62 transfer attempts: 29 transferred, 2 with the reply lost, 13 planned abandons, 18 "not leader" | Porcupine | Ok. 16 operations over 9 s in two stall episodes, cause not established | tests/m0/results/m0-wp4.md, "Chaos run on graft-wp4"; raw/raft-chaos-wp4-cf-1.json |
wp4-cf-5 | B0-11 and B0-06 rules (STEP_DOWN count 6, quarantined target) | Cloudflare, graft-wp4-*, 2026-10-30T07:25Z to 09:30Z, 8 clients from the orb | 2 h 5 min | 184,349 (3 failed at the orb's own fetch); p50 246 ms, p99 1,003 ms | 343 resets. 253 transfer attempts: 119 transferred, 1 with the reply lost, 51 planned abandons, 82 "not leader". 14 slow-storage periods. Call faults on for 39 of 125 min | Porcupine; STEP_DOWN trips from 22,979 member log lines | Ok, 0 operations over 9 s. 23 confirm trips, all on a slowed site | tests/m0/results/m0-wp4.md, "Two-hour chaos run on the final code"; raw/raft-chaos-wp4-cf-5.json |
lin24 | BM-1, 24 h | Cloudflare, graft-m1-lin-*, repo:915635, 2026-11-01T18:27:14.621Z to 2026-11-02T18:27:14.621Z | 24 h | 2,739,691, 0 with no reply; p50 326 ms, p99 1,009 ms. 1,096,016 gets, 821,524 puts, 47,733 CAS ok, 774,418 CAS refused | 5,547 resets (3,676 of a leader), 3,743 transfer attempts (1,131 planned to be abandoned, 73 accepted by the leader, 3,663 refused by the target), 77,653 failed calls, 72,130 dropped AppendEntries replies, counters from 88 isolates | Porcupine -seg 2000 per key, recorded clocks Illegal on k2, k4, k5. Widened 100 ms Ok on all 6 keys. Widened 275 ms (the measured bound, halved) Ok on all 6 keys. No member-side ack evidence. Lost-write audit of all three logs | Not passed. The 5 Illegal segments read writes whose history rows were lost in 5 of 12,518 client chunks, and are Ok with the rows restored. 0 of 869,257 acked writes lost, 0 of 774,418 refused CAS in the log, three identical logs of 479,965 entries. An early ack is not ruled out: one ordering needs 93 ms of ATL to LHR skew against a measured 49 ms, and 17 writes were acked faster than a quorum could commit them on their own client's clock (F44) | tests/m1/results/m1-wp2.md, "The 24-hour run's result, 2026-11-02: lin24 does not pass"; tests/m1/lin/raw/lin24.report.json, lin24.events.jsonl, lin24.jsonl |
lin24b | BM-1, 24 h | Cloudflare, graft-m1-lin-*, repo:934783, 2026-11-03T00:55:07.302Z to 2026-11-04T00:55:07.302Z | 24 h | 2,850,523, 0 with no reply; p50 314 ms, p99 952 ms. 1,141,028 gets, 854,483 puts, 49,811 CAS ok, 805,201 CAS refused | 4,176 resets (2,795 of a leader), 2,752 transfer attempts (855 planned to be abandoned, 56 accepted by the leader, 2,690 refused by the target), 83,209 failed calls, 77,604 dropped AppendEntries replies, counters from 97 isolates | Porcupine -seg 2000 per key, recorded clocks Illegal on k0, k1, k2, k3, k4. Widened 100 ms Ok on all 6 keys. Widened 153 ms (the measured bound, halved) Ok on all 6 keys. Ack check Ok on 904,294 of 904,294 acknowledged writes, 0 violations. Lost-write audit of all three logs. Single-clock early-ack test | Pass. Of the 20 recorded-clock Illegal segments, 18 read or hold writes whose history rows were lost in 15 of 12,535 client chunks, and are Ok with the rows restored. 2 are cross-colo ties, Ok with 1 ms (LHR–SIN) and 4 ms (ATL–LHR) of skew against that minute's measured 92 ms and 63 ms. None is same-colo. 0 of 904,294 acked writes lost, 0 of 805,201 refused CAS in the log, three identical logs of 513,232 entries, no request id twice. Early-ack test 0. F44 answered: a client-clock artifact, since 59 writes acked faster than their own quorum proof's round trip on the client's clock all pass the member check | tests/m1/results/m1-wp2.md, "The 24-hour rerun's result, 2026-11-04: lin24b passes"; tests/m1/lin/raw/lin24b.report.json, lin24b.events.jsonl, lin24b.jsonl |
Other gate runs on Cloudflare, Milestone 1
| Gate | Runs | Load and faults | Check | Result | Source |
|---|---|---|---|---|---|
| BF-12 | bf12-full, 2026-11-01 02:09Z to 02:44Z, 10 groups | 10,001 leader kills sent, 10,000 landed: after_quorum 4,800, after_quorum_unsynced 4,800, each plus 200 with the object held down 12 s | Re-send under the same RequestId; audit of all 30 members | Pass. 10,000 of 10,000 kept their outcome, 0 double applies, 76 answered by another leader | tests/m1/results/m1-wp2.md, "BF-12: leader killed after quorum and before the reply"; tests/m1/wp2/raw/bf12-full.bf12-audit.json |
| BF-12 | bf12-p3 regression, 2026-11-01 | 1,000 kills, 1,000 landed | Same audit | Pass, 0 changed, 0 double applies | tests/m1/results/m1-wp2.md, "BF-12 and BF-13 regression"; tests/m1/wp2/raw/bf12-p3.bf12-audit.json |
BF-12 on Push | p2, 2026-11-01, 10 repositories | 1,000 kills through the git edge's commit_hedged | Re-send outcome and per-ref result; audit of 30 members | Pass, 1,000 of 1,000, 0 double applies | tests/m1/results/m1-wp6.md, "BF-12 on Push commands"; tests/m1/integ/raw/p2.bf12-report.json |
| BF-13 | bf13-full (400 rounds), bf13-p3 (400), bf13-p3e with early AppendEntries (120) | Leader killed before commit on an entry with a Stale refusal caused by an earlier command in the same entry | Porcupine on 1,600, 1,600, and 480 operations; a planted refusal made the bf13-full and bf13-p3 histories Illegal | Pass. 0 refusals returned from uncommitted entries, Porcupine Ok on all three | tests/m1/results/m1-wp2.md, "BF-13" and "BF-12 and BF-13 regression"; tests/m1/wp2/raw/bf13-*.bf13-report.json |
| BF-14 | r1 11:39Z and r2 13:03Z, 2026-11-02, hosted edge | 1,200 commands each (600 Push, 600 PutAccount), every first receipt missed: half held 2.6 s, half dropped (test-faults) | Edge trace against every member's arrival rows | Pass. 2,400 of 2,400 decided ok, 0 forwarded | tests/m1/results/m1-wp8.md, "BF-14: receipts that miss the deadline"; tests/m1/wp8/raw/r1.bf14-report.json, r2.bf14-report.json |
| BF-10 | g1, 2026-11-01 | 1,000 chain creations from two edges at once (13.9 min), then 100 rounds during an owner move, plus a 20-round control without the generation claim | /census of every member | Pass. 1,000 of 1,000 and 100 of 100 with one group. The control made two groups in 20 of 20, which is a design problem found | tests/m1/results/m1-wp7.md, "BF-10: one group per chain, and no read or vote without a config" |
| BF-11 | g2, 2026-11-01, 30 chains | 1,500 cold-cache requests across a member retry, a ReplaceMember stand-in, and an owner move | Every request found the group | Pass, 1,500 of 1,500 | tests/m1/results/m1-wp7.md, "BF-11" |
| BF-15 | g1, then rerun i1 on the integration site script | 1,000 commands queued behind a token revocation, and 1,000 behind an account suspension, per run | Every command decided after the change | Pass, 1,000 of 1,000 refused 401 in each case, in both runs | tests/m1/results/m1-wp7.md, "BF-15" and "BF-15 rerun on this pass's site script"; tests/m1/wp7/raw/i1.bf15.jsonl |
| BF-16 | Stand-in group, then the real groups, 2026-11-01 | N+1 edge with an unenabled format; forced unknown and known-new formats | Member receive counters; refs and entries unchanged | Pass on both | tests/m1/results/m1-wp6.md, "BF-16 on Cloudflare" and "BF-16 on the real groups" |
| BF-17 | Stand-in group (git 2.56.0 and 2.39.5), then the real groups | Atomic push of 10,000 tags with a 1 MiB pack; 10,001 tags; 12,000 non-atomic creates over the byte limit | Entry size, refs, clone, fsck | Pass, 12 of 12 rows on the real groups | tests/m1/results/m1-wp6.md, "BF-17 on Cloudflare" and "BF-17 on the real groups" |
| BF-18 | Bed with a stubbed chain leader: networks 5000001 to 5000003 and the LRU rerun 5000020. Hosted with a real ChainLog: run 1 and run 2, 2026-11-02 | Archive a range, kill both shards, rehydrate, a second shard behind its cursor | git fsck --strict, 9 clones per run, a fetch | Bed: 4 runs, 0 failures. Hosted run 1 failed at rehydrate: a thin stored pack in the promoted range, now a design rule. Hosted run 2 Pass: fsck clean, 9 of 9 clones | tests/m1/results/m1-wp5.md, "BF-18 on Cloudflare Containers", "BF-18 rerun after the LRU", "Run 1" and "Run 2" |
| BF-19 | r10 and r11 on real buckets with a stubbed request table; r1 on the hosted cron | 1,000 failed placements plus 1,000 commits with the marker write killed per run (r10 and r11 reached 2,150 of each with retried batches) | HEAD of every pack and marker in all three buckets after each sweep; injected sweeper clock | Pass. 0 committed packs deleted, every missing marker repaired, every failed placement deleted after 24 h and none before | tests/m1/results/m1-wp4.md, "BF-19 on real buckets" and "BF-19 on the hosted cron"; tests/m1/wp8/raw/r1.bf19-report.json |
| B0-01 on the Rust host | b1 to b6 (first pass) and d1 to d6 (second pass), 60 s each | 6,000 commands per run, 0 failed | Commit p99 on the leader, against M0 controls in the same hour | First pass p99 645 to 791 ms in all 6. Second pass p99 414 to 807 ms: 2 of 6 pass 600 ms, 4 between the lines, none over 1 s. On the current host (m0-carried, graft-m3c, 2026-11-11): 6 of 6 at p99 244 to 377 ms, Pass (tests/m0/results/carried.md) | tests/m1/results/m1-wp2.md, "Commit latency, B0-01 style" and "B0-01 on the new host" |
| F43 soak | f43-soak, 2026-11-02 19:28Z to 20:21Z, 4 repositories | 14 cycles: 4 total faults on a follower site, 4 gray faults (30%) on the leader's site, 3 follower resets, 3 without chaos | Push results; follower last against the leader 60 s after each fault | 374 of 374 pushes ok. Found a different defect, F45: restore ran out of memory on a log of large entries, since fixed | tests/m1/results/m1-f43.md, "The soak"; tests/m1/results/raw/f43-20261102/soak/summary.txt |
Git conformance on Cloudflare (BM-1, git half)
BM-1's fault is B0-03's simulated region loss: every call to and from site 0's member class fails from 120 s into each t/ run, or 60 s into each clients.sh run, to the end. The gray rerun fails each such call with chance 0.3 instead.
git's own smart-HTTP tests, 16 scripts from t/t55xx to t/t58xx, counted from each run's results.tsv:
| Run | git | Fault | Tests | Pass | Fail | n/a (hosted) | Skip | Baseline passes lost |
|---|---|---|---|---|---|---|---|---|
conformance-20261101 | 2.51.0 | none | 535 | 413 | 15 | 99 | 8 | baseline |
conformance-20261101 | 2.56.0 | none | 532 | 412 | 15 | 97 | 8 | baseline |
t-251, 2026-11-02 04:03Z to 04:49Z | 2.51.0 | site 0 failing | 535 | 415 | 14 | 98 | 8 | 0 |
t-256, 05:51Z to 06:33Z | 2.56.0 | site 0 failing | 532 | 412 | 14 | 98 | 8 | 1 (t5549 #2, a test that sets a server-side receive.hiderefs) |
t-239, 06:33Z to 07:15Z | 2.39.5 | site 0 failing | 491 | 377 | 16 | 90 | 8 | no 2.39.5 baseline. Against t-251, only t5541 #20 and t5549 #2 pass on 2.51.0 and not here, both from fixtures the host cannot reproduce |
t5541-nofault, after the t5541 #20 fix | 2.56.0 | none | 42 | 30 | 3 | 6 | 3 | #20 now passes |
| Runs | Fault | Result |
|---|---|---|
| Stand-in group: 2.39.5, 2.51.0, 2.56.0 | none | 29 of 29 each |
| Real groups: 2.39.5, 2.51.0, 2.56.0 | none | 29 of 29 each |
clients-239, clients-239-homed, clients-251, clients-251-homed | site 0 failing | 29 of 29 each |
clients-239-gray, clients-251-gray, 2026-11-02 13:10Z to 13:22Z | site 0's calls failing 30% | 29 of 29 each. Every leader move was an election, and no STEP_DOWN fired on any member |
SSH git sessions
Each session is one git clone, fetch, or push over SSH through a Cloudflare Tunnel to cloudflared in three containers. Two workers ran one session every 6 s each.
| Run | Gate | Where and when | Sessions | Succeeded | Faults | Pushes | Result | Source |
|---|---|---|---|---|---|---|---|---|
b0-24 | B0-24 | Cloudflare, Pro zone, 2026-10-30T23:58:59Z to 2026-10-31T02:10:49Z (2.2 h) | 2,637 | 2,634 (99.886%) | 13 container kills | 878 acked, all in the group log | Between the lines (pass 99.9%, fail 99%) | tests/m0/results/m0-wp7.md, "Results (run b0-24 ...)"; tests/m0/results/raw/wp7/b0-24/summary.json |
b0-24a-r4 | B0-24a | Cloudflare, Pro zone, 2026-10-31T07:36:12Z to 2026-11-01T07:36:07Z (24.0 h) | 28,347 | 28,230 (99.587%) | 143 container kills, one every 10 min | 9,418 acked, all 9,418 in the group log | Between the lines. 85 of 117 failures within 2 min of a kill, the rest Tunnel-edge drops. Carried to M6-WP2 | tests/m0/results/m0-wp7.md, "The run of record: b0-24a-r4"; tests/m0/results/raw/wp7/b0-24a-r4/summary.json |
r1 | none (the plain-SSH relay the operator allowed until go-live) | Oracle relay to the same Tunnel, 2026-11-03T00:54:33Z to 01:19:25Z (24.9 min) | 249 relay + 249 cloudflared control | 249 and 249 | One forwarder restart: sessions open at the stop were cut, new ones worked within about 0.4 s | 166 acked, all in the group log | 0 failures, which cannot tell this path from M0's 99.86% floor at this sample size | tests/m1/results/oci-ssh.md, "Measurements (run r1 ...)"; tests/m1/results/raw/oci-ssh-20261103/r1/summary.json |
Other Milestone 0 fault and load rows on Cloudflare
These rows ran on the M0 prototype. Verdicts are the ones build/BUILD-PLAN.md, "Milestone 0 status", carries, or the results file's where that table has no row.
| Row | Runs | Load and faults | Result | Source |
|---|---|---|---|---|
| B0-01 | 23 one-minute runs, one entry in flight | 81 to 100 commands/s, 1 KB | p99 399 to 690 ms, median 491 ms. Pass in 19 of 23 at the restated 600 ms line, none over 1 s | tests/m0/results/m0-wp2.md, first table |
| B0-02 | 3 reruns at 300 commands/s offered | Group commit, one entry in flight | 159 to 280 commands/s carried at p99 750 to 1,288 ms. Between the lines, carried forward. Rerun on the Rust host (2026-11-11): 6 of 8 runs at 308/s carried at p99 286 to 509 ms, 2 between. Second rerun with the STEP_DOWN fix (2026-11-12): 12 of 16 at 276 to 579 ms, 4 between (678 to 790 ms), every miss on a group with a near follower over 120 ms. Between on every-run reading, F68. Third rerun on groups placed within NEAR_RTT (2026-11-12): 15 of 16 at 249 to 540 ms (8 of 8 at 300 offered, 7 of 8 at 330), 0 failed, and 1 between at 614 ms from one leader stall. Between. Fourth rerun with streamed member-call bodies (m0-carried-4, 2026-11-12): 11 of 16 at 342 to 588 ms, 5 between (614 to 738 ms), then 3 of 4; every miss follows a slow leader storage confirm (p99 132 to 399 ms) or one follower stall. Between. Fifth rerun, a same-window A/B of streamed and array bodies (m0-carried-5): 5 of 8 at 330 with each (361 to 911 ms), so the stream is not the cause; every miss is an AppendEntries held until the leader's own write confirms. Between, carried forward | same; tests/m0/results/carried.md; tests/m0/results/carried-2.md; tests/m0/results/carried-3.md |
| B0-03 | 26 runs (13 failing-calls, 13 blackholed) plus 6 reruns on the final code | One site's member class failing or blackholed, about 100 ops/s | RTO 11.0 to 12.9 s, 0 failed operations. Pass at 13 s | same, and "B0-03" |
| B0-06 | Three modes, 2 min each, plus reruns | 30% of one site's calls failing; 30% of its storage writes failing; only the edge's calls to it failing | Every group moved within 20 s. Pass, 2 minutes per mode, not an hour | same, and "B0-06 and STEP_DOWN" |
| B0-18 | 3 runs | 16 learners at 1,000 reads/s each | 16,000 reads/s, 0 failed, learner p99 683 to 743 ms. Pass at the restated 1 s | same |
| B0-25 | 9 runs with the STEP_DOWN hold, 300 pushes each | 22% of member calls and storage writes failing | 92.3% to 98.7% within three attempts, median 95.0%. Pass at the restated 92%. 17 earlier runs on older code gave 79.3% to 98.3% | same, and "B0-25"; tests/m0/results/raw/b0-25-*.json |
| B0-26 | 4 runs | Every member of 4 or 8 groups reset inside 60 s under 100 writes/s | Every group committed within 1.2 s of its last reset. Pass | same |
| B0-04, B0-05, B0-08, B0-09, B0-10, B0-27 | Site days on 300 or 1,000 groups | 100 writes/s; follower regressions, verify refusals, drain guard | Pass. 0 failed writes in every site day | tests/m0/results/m0-wp4.md, first table |
| B0-07 | Runs 11 to 16 on the drain-alarm code | Gray site C during B's site day | 179 of 182 groups moved within 20 s as written, 181 of 182 under the 50 s form. Carried. Rerun on the Rust host (2026-11-12) with test-faults stand-ins for graft-ops's drain and SiteHealth: 235 of 235 within 20 s over 3 days, max 17.2 s, 0 failed writes. Pass on the member path; M5-WP4 reruns it with the real drain | tests/m0/results/m0-wp4.md, "B0-07 on the final code"; tests/m0/results/carried.md, B0-07 |
| B0-22 | 8 rehydrations | 5.38 GB network from R2 into a new container | At most 118.1 s, median 67.9 s. Pass, thin margin | tests/m0/results/m0-wp6.md, "B0-22" |
| B0-23 | 8 runs | 33 CI clones/s, tip moving 10 times a minute | Pass with a push hook (100% from R2, p99 406 ms) or clones 5 to 60 s after a push (97.5 to 97.8%). Up to 44% refused at the instant of a push without the hook | tests/m0/results/m0-wp6.md, "B0-23" |
| B0-29 to B0-35 | Eight passes of the agent-fleet replay | 500 agents, journey 9 | B0-29 passes in 5 of 8 runs. B0-31, B0-32, and B0-33 pass on the final rules. B0-30 passes at the margin (accelerated), and B0-35 passes. B0-34 is a partial pass at reduced scale | tests/m0/results/m0-wp9.md, "Verdicts", "Second-pass summary", and "Eighth-pass verdicts"; build/BUILD-PLAN.md, "Milestone 0 status" |
| L0-12, LF-12 | L0-12 in the first and third to seventh passes, and LF-12 runs r2 to r37 | A1 replay with leases, held feed polls, and leader kills | L0-12's feed clause passes for all five actor kinds in the sixth pass. LF-12 is between the lines: 1 pass and 3 between in 4 runs on the final rules | tests/m0/results/m0-wp12.md, "Sixth-pass summary" and "Tenth-pass summary" |
Simulator
crates/raft/tests/sim.rs runs three members under one seeded generator per run. HARSH drops 8% of messages, duplicates 4%, delays each 5 to 1,500 ms, crashes members, cuts sites off, and skews clocks. GRAY adds one gray site at a time: failing calls, 1 s storage confirms, or failing writes. After every message and step the simulator checks one leader per term, identical committed entries, two disks per committed index, every Acked write in every later leader's log, no RequestId at two indexes, BF-14's no-forward rule, and the ack evidence of every acked write (ack_check).
The long sweep, run on 2026-11-03 at c0c6fb9 with RAFT_SEEDS=2000 RAFT_STEPS=10000 in release mode. It ran twice with identical counts, in 344 s and 347 s:
| Half | Steps | Operations | Acked writes | Faults | Porcupine on the exported history | Result |
|---|---|---|---|---|---|---|
| HARSH | 20,109,261 | 160,066 (49,629 acked, 18,649 refused, 60,710 reads, 31,078 no reply) | 49,629, all with valid evidence | 226,083 crashes (97,866 mid-input), 31,230 partitions, 696,535 of 2,215,368 messages dropped, 3,524 leaders | Ok on 156,723 operations | Pass, 0 forwarded |
| GRAY | 20,082,543 | 2,226,291 (951,227 acked, 384,638 refused, 890,206 reads, 220 no reply) | 951,227, all with valid evidence | 17,108 gray faults, 24,327 write failures, 15,329 partitions, 1,534 STEP_DOWN transfers, 12,727 leaders | Ok on 2,226,280 operations | Pass, 0 forwarded |
Unit and property tests
cargo test --workspace at c0c6fb9, 2026-11-03: 259 passed, 0 failed, 2 ignored, in 25 test binaries.
| Crate | Tests |
|---|---|
graftyard (cli) | 20 |
graftyard-fold | 9 |
graftyard-git | 87 |
graftyard-identity | 39 |
graftyard-lookup | 1 |
graftyard-raft | 76, plus long_sweep and long_liveness ignored by default |
graftyard-release | 13 |
graftyard-site | 3 |
graftyard-types | 11 |
graftyard-edge, graft-m1-wp4, graft-m1-wp5, graft-m1-wp6-bed | 0 |
| Total | 259 |
Planted bugs
A planted bug is a code mutant applied alone, run against a test suite, and then reverted.
| Where | Planted | Caught | Source |
|---|---|---|---|
crates/raft, M1-WP2 first pass | 8 | 8 | tests/m1/results/m1-wp2.md, "The tests catch real bugs" |
crates/raft, M1-WP2 second pass | 4 | 3. Bug D, a resumed led_term sending its Noops before they are durable, was not caught | tests/m1/results/m1-wp2.md, "The simulator" |
crates/raft, Member::record | 1 | 1 | tests/m1/results/m1-wp2.md, "Gate-row pass, 2026-11-02" |
crates/raft, the ack mutants A and B | 2 | 2, by the ack check under GRAY (54 of 3,873 acked writes flagged) | tests/m1/results/m1-wp2.md, "The planted mutant" |
crates/raft, M1-WP3 | 17 | 17 | tests/m1/results/m1-wp3.md, "The tests catch real bugs" |
crates/git, M1-WP4 pack store and sweeper, plus M1-WP5's alarm | 25 | 25, after one plant was rewritten to compile and one miss led to a new assertion | tests/m1/results/m1-wp4.md, "Adversarial unit tests and planted bugs" |
M1-WP5: the shard server's LRU (2) and sweep_all (2) | 4 | 4 | tests/m1/results/m1-wp5.md, "The LRU of bare repositories" and "The sweeper's cron entry" |
crates/git, M1-WP6 | 6 | 6 | tests/m1/results/m1-wp6.md, "Planted bugs" |
crates/identity, M1-WP7 | 18 | 18 | tests/m1/results/m1-wp7.md, "Planted bugs" |
| Total | 85 | 84 |
Smoke, local, and discarded runs
None of these runs counts toward a total above.
| Kind | Runs | Counts |
|---|---|---|
| Smoke runs on Cloudflare, Porcupine Ok | cf1 (150 s), cf-remote-smoke (180 s), cf-remote-2 (300 s), cf-boot-smoke (10 min), lin-smoke1 (9 min), lin24b-smoke (30 min) | 112,985 operations: 2,729, 6,658, 11,188, 19,030, 16,243, and 57,137. lin24b-smoke's ack check was Ok on 18,225 of 18,225 acked writes |
Local chaos runs on wrangler | 32 raft-chaos-local* runs | 744,258 operations, all Ok. Local workerd, not Cloudflare |
| Discarded as set up wrong | wp4-cf-2, wp4-cf-3, wp4-cf-4; conf-run2 (stale shard containers) | Not checked or not kept |
| Stopped 24-hour SSH attempts, each on a defect since fixed | b0-24a-24h, b0-24a-r2, b0-24a-r3 | 1,144 sessions: 341 (338 ok), 536 (533 ok), and 267 (263 ok) |
Stopped t/ run | t-256-first | Found fix 4 in m1-wp8.md, then rerun as t-256 |
build/BUILD-PLAN.md
Milestone 0: measure the platform and the protocol
Milestone 0 status (2026-10-31)
Every package ran on prototype code in the one Milestone 0 account, M0-WP6 and M0-WP7 on its one Pro zone, graftyard.dev. Results are in tests/m0/results/, one file per package, and each fix the runs found went into the design first (decisions/DECISIONS.md, "Milestone 0"). No row hit its Fail value. Milestone 0 is closed for starting M1 (FRAME.md, "Operator decision, 2026-10-31, Pro only and no GHES"; decisions/DECISIONS.md, "Decisions for a Pro-only build with no GHES, 2026-10-31", "M1 starts with the between rows carried forward"): the rows below carry forward to the owner named, with no gate number changed, and a Fail value on any of those reruns still sends the design back. Rows that need a plan, an instance, or accounts the operator does not have stay not run, owned by whoever first has them, and gate no M1 work. Rows still short of their Pass value, each with its owner:
| Row | State | What closes it |
|---|---|---|
| B0-11 | Passes: 0 violations on 5,742,736 operations and 0 lost acknowledged writes in 48 hours, with a 100 ms clock allowance across client colos that the run did not measure; fault coverage on site C is thin (F36). Rechecked with the fixed segmented checker, the verdict is unchanged at 0 ms and 100 ms: none of its 4 no-reply writes was ever observed, so dropping them, as M0's check did, changed nothing here, though it is not "only stricter" in general (tests/m1/results/porcupine-seg.md). Its 24-hour rerun on the M1 code, lin24, had fault counters in all 88 isolates and recorded clocks, and did not pass, because an early ack was not ruled out; the second rerun, lin24b, passed with member-side ack evidence, which answered that as a client-clock artifact ("Milestone 1 status", BM-1; decisions/DECISIONS.md, "F44 answered: the fast acks were a client-clock artifact") | Done (lin24b) |
| B0-01 | Passes on the Rust host (m0-carried, graft-m3c, 2026-11-11): 6 of 6 one-minute runs at 100 commands/s, commit p99 244 to 377 ms, 0 failed; M1-WP2's host had passed 2 of 6 (tests/m0/results/carried.md) | Done |
| B0-02 | Passed on m0-carried-8 (graft-m3d, standard-4 shards, 2026-11-13, tests/m0/results/carried-8.md): 8 of 8 at 300 and 8 of 8 at 330, commit p99 249 to 579 ms, on main 84d29c2 (F67 rules 10 to 13) plus an AppendEntries pipeline (PIPELINE 4, stopped only on a lossy link), the LOOP_LAG step-down, ENTRY_SOFT at 100 KB and APPEND_BYTES_MAX at 100 KB. Heavy turns stayed at the 60 KB tree's level (28 /commit turns of 80 ms or more over 16 runs against 24; CPU p999 23 to 49 ms), and confirm p99 was 108 ms at 80 to 100 KB and 131 ms at 100 to 130 KB. Long sweep (live half 4,000 seeds at an absolute 30 s), Porcupine, long_liveness, plants 15 of 16 and the full checks pass. Before the two size changes the same code passed 12 of 16. Open: one run's LOOP_LAG trip moved the leader to weur, where the apac voter's link was lossy at these call sizes; and LM-1's B0-29 A/B on this tree before merge | Merge after LM-1's journey 9 A/B |
| B0-07 | Passes on the Rust host (m0-carried, graft-m3c, 2026-11-12), with test-faults stand-ins for graft-ops's drain and SiteHealth, which the hosted code does not have yet: 3 days, C detected 3.6 to 3.7 s after the fault, 235 of 235 groups led on C moved within 20 s (max 17.2 s), 0 failed writes, never two sites drained (tests/m0/results/carried.md). M0's one group at 20.2 s stays unexplained for the prototype | Done for the member path; M5-WP4 reruns it with the real graft-ops drain |
| B0-20, B0-21 | Ran on the Pro zone graftyard.dev: the 100 MiB cap holds for sized and chunked bodies, and git over it sees a bare 413, as the 413 decision accepts. Business and Enterprise not run; their caps (200 MB, 5 GB) stay doc | The first install on that plan |
| B0-24a | Between pass and fail: 99.587% of 28,347 sessions over 24 h with 143 kills; 85 of 117 failures within 2 min of a kill, the rest Tunnel-edge drops (background near 99.89%); 0 acknowledged pushes lost. The option stays built, off by default. Gates only M6-WP2's SSH option, not M1 | M6-WP2's rerun on real code |
| B0-24b | Enterprise Spectrum: not run (Spectrum refuses a Tunnel origin on Pro); the ingress is designed and not built | The first Enterprise install |
| B0-29 | Passes in 5 of 8 runs; misses at 602 and 625 ms track placement (first pass, r2 and the sd3r1 variant) and 809 ms tracks storage stalls on the leader (fifth pass, p5r1) | LM rerun on real code |
| B0-34 | Partial pass at reduced scale, with the composed-miss rules now in the design | A full-scale run with the warm pool |
| L0-06, L0-11, L0-04 | Partial: predictor without other git work; byte diff without a GHES oracle, which becomes the pinned ghes-3.22 descriptions and github.com inference; agent CI without an Actions engine | LM rows that rerun them on real code (LM-4, LM-1, LM-3) |
| L0-07 | Vendor cells not run: they need vendor accounts and a public host; the documented-behavior drivers are the evidence (Codex's documented permissions fail the check cells). Does not block M1; no LM gate names it | The operator's vendor accounts |
| L0-08 | The ghr_ cell needs a GHES 3.22 instance, and none exists. AM2-WP1 builds A4's fallback, so the cell becomes the held-token refresh and A4 gates nothing | LM-2's rerun of L0-08 on real code (AM2-WP1) |
| LF-12 | Passes on real code after am1-fix (lf1-6, AM1-WP6 status): leader commit p99 474 ms placed and 710 ms before, 5,000 of 5,000 polls held on 11 learners and 0 on voters, and 0 failed AppendEntries calls of any kind in 165 /stats samples, so zero overloaded (tests/m0/results/carried.md, LF-12). On the prototype it was 1 pass and 3 between in 4 runs | Done |
M0-WP1. Test bed, bindings, and placement facts
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| B0-12 | Member placement: 1,000 members created with each site class's hint, median of 20 binding round trips to each other voter | At least 95% of voter pairs at 30 ms or more and consistent with the hinted regions | Under 80% |
| B0-13 | Bucket placement: 20 buckets per hint, location from the get-bucket API against a latency probe | 100% agree | The API reports the hint, not the placement |
| B0-15 | Idle cost: 100,000 idle groups for 24 hours across a UTC midnight, with the daily Rtt recheck shortened to hourly, beside 1,000 control groups that serve one request an hour | Zero duration billed and zero Rtt checks for idle groups, while the control groups show both | Any duration billed, or any Rtt check, for idle groups, or a control group that shows neither |
| B0-16 | Account-owned deploy token with Workers Scripts Edit and Containers Edit runs a deploy with a container image | Deploys | Fails, which moves the deploy token to a user token held by the customer |
| B0-38 | Shard containers under the durable_object policy: 30 starts from objects created with each site's hint, then a new site-script version with a changed image map while three run | Every container in its site's container regions, and no running container restarts or changes image | A container outside its site's regions that the start check misses, or any restart on deploy |
M0-WP2. Raft prototype on Durable Objects
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| B0-01 | Commit p99 on Tri, leader enam, 1 KB commands, 100 commands/s, member calls through cross-script DO bindings | At most 600 ms | Over 1 s |
| B0-02 | Group commit on one group, one entry in flight | At least 300 commands/s at commit p99 at most 600 ms | Under 100 commands/s |
| B0-03 | Write and read RTO, with one site script's member class failing every call (simulated region loss) under 100 writes/s, run once with every client in one colo and once with calls to that class blackholed (hung, not failing) | New leader committing and serving fresh within 13 s at p99 in both runs | Over 60 s in either run |
| B0-11 | Linearizability: 48 hours of Porcupine-checked histories while member calls to one site are dropped at random, members are reset, and leadership transfers, some abandoned mid-way | Zero violations, zero lost acknowledged entries | Any one |
| B0-25 | Partial DO failure: inject 22% errors into member calls and into storage writes, which reset objects, and count whole pushes | At least 92% of pushes succeed within three attempts | Under 84% |
| B0-26 | Reset storm: every member on all three sites reset within 60 s under 100 writes/s, with suspect reports from two colos | Every group commits within 12 s of the last reset | Any group over 12 s |
M0-WP3. Gray failure and learners
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| B0-06 | Gray site, with 30% errors injected on one site's member calls, then separately on its storage writes | Every group led there moves within 20 s and changes leader at most twice in an hour | Any group slower, or more than two changes |
| B0-18 | Learners: 16 learners serving 16,000 reads/s on one group | Leader under 700 req/s, learner read p99 at most 1 s | Leader returns overloaded |
M0-WP4. Site days, verify refusals, and drain
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| B0-04 | Site deploy under 100 writes/s, with groups led from every site and clients on the leaders' continents: drain, deploy at 1%, 10%, 50%, 100%, undrain | Zero failed writes, commit p99 at most 1 s for clients on the leader's continent | Any failed write |
| B0-05 | Follower regression: a release whose N+1 followers throw while applying one event kind | The apply gate stops it at 1% and graft-ops runs the pre-signed rollback, with no failed write and no human signature | It reaches 10%, or any write fails |
| B0-07 | Gray site C during B's site day at 10%, with automatic drain on | Callers on A and B report C, graft-ops undrains B under the 1% lead limit and drains C. Every group led on C that served a request or committed an entry in the 50 s before the drain moves within 20 s of it, and every other group led on C moves within 20 s of its next request | Two sites drained at once, or any group on C slower |
| B0-08 | verify refusal by an older follower: release N+1 on site A writes a shape N cannot read | Every group A led moves to an undrained follower holding its last index; other writes keep committing; A stays non-leader | Any group without a leader for more than one lease |
| B0-09 | Rolling site refuses: N+1 followers on B refuse valid N entries at 10% of a site day | No leader steps down, B is marked unable to follow, an alarm fires, and the site day halts and rolls back | Any group without a leader for more than one lease |
| B0-10 | Two-site residual: A and B on N+1 commit a shape C on N cannot read | Each group keeps its leader and commits on A and B, C is marked unable to follow, and the site day halts | Any group without a leader for more than one lease |
| B0-27 | Drain guard: two drain-key SiteMode { Drain } entries for different sites submitted straight to the Release group, then a test group whose voters are all drained by customer-key entries | The second drain-key entry is refused and pages, and the all-drained group elects a leader within 12 s | Two sites drained by the drain key, or the all-drained group leaderless over 60 s |
M0-WP5. Account-separated link
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| B0-17 | Account-separated option: member RPC over signed HTTPS through the site Worker, against a cross-script binding | p50 overhead at most 20 ms | Over 60 ms, which withdraws the option |
M0-WP6. Git on Containers and R2
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| B0-14 | Daily working set of git cache shards per tier, in bytes, on a synthetic replay | Recorded; cost.py shard counts updated | None. The result sets the shard count |
| B0-19 | Container starts with the control plane out of reach: 100 new instances started with every account API token revoked | 100 of 100 start | Any fails; the control-plane row stays contained either way and warm shards stay sized for a day |
| B0-20 | Push size: 2 GB push through the edge on Enterprise, teed to two buckets and a container; on Pro and Business, a push over the edge cap | Enterprise completes with Worker memory under 100 MB; Pro and Business print the edge's report-status naming Enterprise or splitting the push | Fails, needs buffering, or shows a bare 413 |
| B0-21 | Chunked push on Pro, Business, and Enterprise | Recorded either way | None. The result sets each plan's stated cap |
| B0-22 | Container rehydration of a 5 GB network from R2 | At most 120 s | Over 600 s |
| B0-23 | Clone cache: 33 CI clones/s of a monorepo whose tip moves 10 times a minute | Hit rate at least 95%, time to first byte p99 at most 500 ms | Under 80% |
M0-WP7. SSH chain
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| B0-24a | The Pro and Business SSH option: clients through cloudflared access ssh as their ProxyCommand to a Tunnel public hostname of type ssh, then cloudflared in three containers, with writes through the outbound handler and one container killed every 10 minutes for a day. On Pro, 2,634 of 2,637 sessions (99.886%) over 2.2 h with 13 kills, then 28,230 of 28,347 (99.587%) over 24.0 h with 143 kills: between the lines. 85 of the 117 failures came within 2 min of a kill, and the other 32 were Tunnel-edge drops (stream closed before the banner, no banner within ConnectTimeout, 90 s timeouts), a background near 99.89%. Failures are dominated by sessions in flight on the killed replica, so a 10-minute kill period puts a floor of about one failure per kill under the rate. 9,418 pushes acknowledged, all 9,418 in the group log [measured, tests/m0/results/m0-wp7.md, B0-24a; tests/m0/results/raw/wp7/b0-24a-r4/summary.json]. Carried to M6-WP2's rerun on real code | At least 99.9% of sessions succeed | Under 99%, which leaves Pro and Business HTTPS only |
| B0-24b | Enterprise native SSH: Spectrum on port 22 to the Tunnel's virtual network, then the same containers, handler, and kills for a day. Not run: it needs an Enterprise zone. On Pro the Spectrum API refused a Tunnel origin (10012, 13002) measured, same | At least 99.9% of sessions succeed | Under 99%, which drops native SSH on Enterprise |
M0-WP8. Restore window
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| B0-28 | Restore window: delete a branch and force-push another on day 1 with GC and archiving running, then restore both to day 0 on day 30 (clock-skewed replay) | Both refs return to their day-0 targets, every object they reach is readable, and archives are named by committed Archived records in two buckets | The restore is refused, any object is missing, or a follower truncates past a range with no committed Archived |
M0-WP9. Agent load on the base
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| B0-29 | Agent fleet replay (journey 9) on one Tri group, leader enam. 500 agents push distinct refs every 2 min (40 KB mean, 5% at 1 MiB), 27.6 commands/s with PrSync, test-merge records, check runs, task updates, and lease renewals, 45 feed polls/s on followers and learners, 10 merges/min, 33 ref reads/s, 200 API reads/s. Commit p99 on this row is set by storage stalls on the leader: of the seven fourth-to-seventh-pass runs, the five with 0 to 4 failed AppendEntries in 600 s passed at 378 to 531 ms, and the two with 10 and 12 missed at 798 and 809 ms [measured, tests/m0/results/m0-wp9.md, fourth to seventh passes]. The line stays one 600 s run at 600 ms, and M1-WP2 reruns it on real code | Commit p99 at most 600 ms, merge p99 at most 2 s, zero Stale on distinct refs, leader under 600 req/s with zero feed polls, API read p99 at most 2 s with at least 99% served ("Follower read") | Commit p99 over 1 s, any queue merge waiting over 10 s, any feed poll on the leader, or API read p99 over 5 s |
| B0-30 | Hot-group storage. 30 days of the journey 9 replay on a group seeded with a 4 GB metadata tier | Member SQLite under CHILD_GROUP_AT with zero SQLITE_FULL, and archived entries readable for RestoredTo | Any SQLITE_FULL, or member SQLite over CHILD_GROUP_AT without a child group taking the tier |
| B0-31 | Same replay with the leader on apac and every agent push at 1 MiB, INLINE_MAX_HOT on, then with every bucket's placement failing, under the design's STEP_DOWN with no variant. The placement-failing half gates Human and MergeQueue latency, not the leader's site: the apac leader may move once on the Human packs that the outage rule keeps inline. On the final Milestone 0 code both halves passed: buckets up, the apac leader held 600 s with commit p99 598 ms and every agent push staged; placement failing, Human and MergeQueue p99 1.11 s with 0 of 287 refused, 2,500 of 2,500 agent pushes refused at once in the ng shape, and one STEP_DOWN from apac at 8.9 s (6 and 3 failed calls) [measured, tests/m0/results/m0-wp9.md, B0-31, eighth pass] | Commit p99 at most 1 s with buckets up. While placement fails, Human and MergeQueue commit p99 at most 3 s and 0 of them refused, and agent pushes over INLINE_MAX_HOT refused at once in the ng shape | Over 3 s, or any Human or MergeQueue push up to 1 MiB refused while placement fails |
| B0-32 | Burst of 500 agent pushes of 256 KiB each in 1 s with merges queued, admission on. The burst rides a hot group, so the row tests INLINE_MAX_HOT, which stages each 256 KiB pack before AGENT_INLINE_ENTRY is reached; a burst on a quiet group is covered by the budget's B0-33 evidence (fresh groups' first entries capped at 128 KiB, seventh pass). On the final Milestone 0 code 505 of 505 burst pushes committed within 5 s (client p99 1.40 s, stage p99 1.19 s), the burst's merges took 129 to 227 ms, each in the next entry (up to 2.30 s at a 256 KiB cap), queued agent bytes peaked at 119 KB, and there were 0 leader changes and 0 suspect [measured, tests/m0/results/m0-wp9.md, B0-32, eighth pass] | Every push acknowledged within 5 s or answered in an HTTP 200 with ng <ref> agent admission full, retry after <n>s, queued agent bytes under AGENT_QUEUE_MAX, every merge in one of the next 2 entries, zero leader changes, and zero suspect reports | Any lost acknowledgment, any other refusal shape, an object reset from memory, a merge delayed over 2 s, any leader change, or any suspect report |
| B0-33 | 5,000 agents on one group, 252 agent-class commands/s offered with caused commands classed, plus one user token writing 100 commands a minute, admission on. With outcome folded into commit, AGENT_INLINE_ENTRY on every group, and hot from the log, three far-flung runs (ATL leader, weur and apac followers, fresh groups) held their leader with 0 suspect, Human and MergeQueue p99 746 to 943 ms, 253 agent commands/s admitted, and the leader at about 500 req/s [measured, tests/m0/results/m0-wp9.md, B0-33, seventh pass]. With commit_fail_min added, a fourth run passed again at Human and MergeQueue p99 674 ms, 250 agent commands/s, and 0 leader changes; no window reached the 5% rate measured, same, eighth pass. Before the fold the leader served 750 to 950 req/s and 2 of 3 runs lost it by election (fifth pass). The zero-leader-change claim is measured for this mix, with a margin of 57 to 326 ms under 1 s, least in the first 30 s | Human and merge-queue commit p99 at most 1 s, at least 140 agent commands/s admitted, the user token admitted as Agent, every refusal in the git ng or REST 429 retry-after shape, zero leader changes, and zero suspect reports | Any leader change or suspect report, any merge-queue command waiting over 2 s, or under 100 agent commands/s admitted, which reopens ref lanes with a parent Abort { txn } record (design/AGENT-LAYER.md) |
| B0-34 | Git CPU for one 5 GB network under the journey 9 replay, with the tip moving at 10 merges a minute along real history, then one warm site killed, then a second network that turns hot with the API unreachable, then a hostile owner making three of its own networks hot beside it | k set in band from NetworkLoad and the hot pool, at most 8 per site, the hostile owner held to its per-owner pool budget, human push p99 at most 3 s beyond transfer, and clone time to first byte p99 at most 1 s throughout | k over 8, any Cloudflare API call, the monorepo's k reduced by another owner, or time to first byte p99 over 5 s |
| B0-35 | Lazy test merges, with the base moving 10 times a minute under 500 open PRs and 500 clients polling their PR every 5 s | Test-merge vCPU within TEST_MERGE_BUDGET, Actions dispatch p99 at most 10 s, and zero computes from list fields | Over budget, or dispatch p99 over 30 s |
M0-WP10. Agent refusals and provenance on the base
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| B0-36 | Agent refusals, run once with a Graftyard-run token and once with a confined vendor App's token, used without the controller. Each tries approve, merge, ready-for-review, a bypass, dispatch, workflow re-run, a release, a label outside the allowlist, a graftyard/ status, a push outside its prefix, a push with no live task and a fork push after its task is cancelled (Graftyard-run token only; a confined App is bound by confine), a REST ref write into another task's Exclusive scope under AgentsExclusive, and a change under the protected list, over git, REST, and GraphQL where GHES 3.22 has a surface for the action. The Graftyard-run token runs once with its shipped permissions and once on an installation widened to every permission, and a refusal from either counts. It also pushes an outsider commit that changes a workflow under its ref, pushes after an approval and then merges, and posts graftyard/agent-gate. Then a flagged App without confine, Graftyard MCP, and a user token with X-Graftyard-Agent-Session push into the same Exclusive scope, and a user token pushes a commit whose only trailer is Co-authored-by | 100 of 100 refused for each, for both agent kinds. The other three are never refused by the lease, and the Co-authored-by push stays Human | Any accepted, any of the other three refused by a lease, or the Co-authored-by push classed Agent |
| B0-37 | Provenance and hidden refs. 10,000 pushes with malformed trailers and notes, forged Graftyard-Session trailers, and writer pushes to refs/graftyard/, then v0, v1, v2, and --mirror fetches | Zero pushes fail on parsing, every forged trailer stays Claimed, every refs/graftyard/ update not by graftyard-workspace is refused, and session refs appear only to a v2 ref-prefix that starts with it | Any push fails on parsing, a forged trailer links, or a session ref is advertised otherwise |
M0-WP12. Leases, the activity feed, and the core tier (layer rows)
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| L0-01 | Gray enam (30% errors) during the A1 replay | Base and fork groups move within 20 s, 99% of agent pushes succeed within three attempts | Any group slower, or under 97% |
| L0-12 | Leases and the feed. The A1 replay with a lease per agent and leader kills, then a 30-minute stall with every member running and member calls blackholed, then an idle 60 s lease followed 10 minutes later by an overlapping Exclusive acquire, then 100 conflicting pushes and 100 conflicting REST ref writes into another task's Exclusive scope under AgentsExclusive, run once per actor kind: Graftyard-run App, confined vendor App, flagged App without confine, Graftyard MCP, and a human user token | No two holders hold overlapping Exclusive leases for longer than DRIFT (500 ms), and measured member clock offsets stay under 500 ms. Every lease live before the stall is live after it, with expires moved by the recorded stall. The late Exclusive acquire succeeds. 500 long-pollers see each event within FEED_BATCH plus 2 s (3 s) at p99 outside leader kills. 100 of 100 refused for each of the two ActorKind::Agent kinds, 0 of 100 for the other three | Any overlap over 500 ms, any lease expired by the stall, the late acquire refused, feed p99 over 2 s, or a refusal count off for any kind |
| L0-13 | Lease limits. Exclusive acquires of **, *, and */x, one with no live task, and a second for one task; an App-token acquire with no session; 600 plan blocks from accounts without push access, then 600 from writers; a holder whose push access is removed; an App uninstalled while it holds leases and tasks; an Exclusive lease renewed for 5 h | 100 of 100 refused for each bad acquire. Zero plan leases from non-writers and at most 500 from writers. The removed holder's lease refuses nothing at the next decide. Uninstall releases every lease and cancels every task. The lease reads as Shared after 4 h | Any accepted, any plan lease from a non-writer, or an Exclusive lease past 4 h |
| L0-16 | Core tier alone. Pushes, minting, and lease checks on the A1 replay with the metadata tier unloaded (a merge reads its pull row and writes Merged, both Metadata, so it waits for the tier) | Every check passes or refuses with zero metadata loads | Any metadata load on those paths |
| LF-12 | A1 replay on one Tri group with no learners, plus 5,000 held feed polls with paths, then again after place_learners runs. On the final rules, with the own-stall exclusion and the all-lagging Retry-After of FEED_BATCH_ALL, 4 runs gave 1 pass and 3 between the lines, with no fail, no step-down, and every other line passing in every phase. The misses trace to Durable Object storage confirm stalls on the leader or the near follower and to the far follower's round trip, as for B0-02 and B0-29 [measured, tests/m0/results/m0-wp12.md, LF-12, tenth pass]. The gate is one run, as B0-29's is | Commit p99, measured at the group's leader (admission to resolution), at most 600 ms once place_learners has run and at most 1 s before it, zero overloaded replies to AppendEntries, at most 1,000 polls held per member with the rest answered 429 with Retry-After, zero polls on the leader, and zero on followers after placement | Commit p99 over 1 s once place_learners has run or over 2 s before it, any overloaded replication reply, any member above 1,000 held, a refused poll without 429 and Retry-After, any poll on the leader, or any poll on a follower after placement |
| LF-13 | 1,000 polls held on a follower, then its member calls failed while commits run and leadership moves to the other follower, so the new leader's first round leaves a gap, then its log truncated past the follower for an InstallSnapshot catch-up (one leader re-sends from the follower's match, so dropped rounds alone leave none, crates/raft/tests/lf13.rs; hosted, the follower's calls are severed until it stands behind the other follower, then leadership moves there, tests/am1/results/lf13-m3c.md) | Every held poll returns within 1 s of the follower's stop, when it learns it is behind, with its cursor and no events, new polls are refused until it catches up, and no event is lost; the time from the fault's lift is reported too (A26, decisions/DECISIONS.md, "LF-13's 1 s runs from the follower's stop") | Any poll held or accepted while it is behind, or any event skipped. Between the lift and its stop the follower cannot know it is behind: it admitted 0 to 2 of 50 probes there, and its stop answered them with nothing skipped [measured, graft-m3c] |
M0-WP13. Token ceilings, tasks, the agent gate, and the trust view (layer rows)
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| L0-03 | Token ceilings with raw tokens, bypassing the router. A workspace token pushes to the base, to another task's prefix, and to .github/workflows/, a token minted with no live AgentTask pushes, a fork-mode token pushes to its fork after its task is cancelled, and the base token merges | 100 of 100 refused for each | Any accepted |
| L0-09 | Trust view. 200 injection payloads (HTML comments, zero-width, bidi, link titles, collapsed blocks) in bodies, comments, commit messages, and check-run text, plus a non-writer edit after assignment | 100% absent from visible and counted, body byte-identical, and the edited task paused | Any payload in visible, or the task runs on the edit |
| L0-14 | Task rules. With agents_enabled off, then on: Graftyard-run creates from an installation token, an AgentApp's user token, an MCP token, a demoted user, and a read-only user; a controller create from an issue assigned to another bot; 101 open tasks for one requester; PATCH of a cancelled task to in_progress; a token request naming another App's task or a repository outside the token's list; a body that sets prefix | 422 for every create while off. When on, 100 of 100 refused for each bad case, the prefix always derived from session, and every create and cancel at the controller within 5 s, at no other App's webhook | Any accepted, or the delivery reaches any other App |
| L0-15 | Gate and read permissions. An MCP client with WriteApproveMerge approves an agent PR. Code Mode calls approve and merge through codemode.request() without that scope. A vendor's named requester approves its PR. Anonymous, metadata: read-only, and over-limit feed polls, and /log and prompt reads with metadata: read only | AgentGate stays unsatisfied in all three cases, and Code Mode is refused 100 of 100. Feed polls get 401, 404 (no contents: read, as task and lease reads answer), or 429, the leader sees zero feed requests, and /log and prompts return 403 | Any gate satisfied, any Code Mode approve or merge, or any read served |
M0-WP14. Sandbox containment and agent CI (layer rows)
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| L0-02 | Sandbox containment, 100 attempts each to reach an unlisted host, read a token from env, files, or /proc, push to another repository on the host with a sandbox-supplied token, create a gist, call GraphQL, write a status or check run, send a model request with mcp_servers or a server-side tool, and fetch an unknown Go module path | 0 succeed | Any succeeds |
| L0-04 | Agent CI under ci: AutoReadOnly on the A1 replay | Agent-push workflows start within 30 s with a read-only token and no secrets, and jobs that reference secrets wait | Any secret reaches an agent-push job before AgentGate holds for its head |
M0-WP15. MCP and unmodified agent clients (layer rows)
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| L0-07 | A2 oracles. Claude Code GHES manifest flow, Cursor's GHES app, Codex's app template, local github-mcp-server default toolsets | Each installs, gets webhooks, opens a PR, reads graftyard/overlap, writes a check, and revokes, with zero Graftyard-specific client code. A push 1 s after revoke is refused 100 of 100 | Any needs a patch, or a revoked token pushes |
| L0-08 | MCP. Three 2026-07-28 clients and one 2025-11-25 client, CIMD, PKCE, consent, 30 calls each | 100% succeed, tool overhead p50 at most 50 ms, 100 of 100 wrong-audience tokens refused, a refreshed token keeps its scope 100 of 100, a replayed refresh revokes the grant, a framed or skipped consent issues no code, and a refresh succeeds with the unscoped ghu_ held, which is deleted after it | Any wrong-audience token accepted, any refresh wider than its grant, or any code without consent |
| L0-11 | Byte diff. 1,000 REST and GraphQL requests from unmodified clients, with and without the agent layer deployed, and the same requests against a GHES 3.22 oracle: with no GHES instance, the pinned ghes-3.22 OpenAPI and GraphQL schema for shape, and github.com for behavior on operations whose descriptions match, weaker than an appliance inference | With and without the layer: identical except request ids and times. Against the oracle: every difference listed, and none in a field a client reads | Any other difference with and without the layer, or an unlisted oracle difference |
M0-WP16. Prediction and provenance (layer rows)
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| L0-05 | Prediction on 1,000 replayed branch pairs of a real monorepo | 100% of textual conflicts on current tips found, at most 5% false conflicts, push to verdict p95 at most 30 s, overlap check on the head SHA before a PR exists | Any missed conflict on current tips, or p95 over 120 s |
| L0-06 | Predictor under the A1 replay, then with its site leading | Second-site shard CPU at most 50%, and zero predictions while that site leads | Over 60%, or any prediction while it leads |
| L0-10 | Provenance. 10,000 replayed agent commits, then a vault restore | 100% resolve to a session or PR before and after | Any lost |
build/BUILD-PLAN.md
Milestone 1: groups and git
Milestone 1 status (2026-11-04)
Milestone 1 is closed: every gate row below is Done, the last being BM-1's 24 h half on lin24b (2026-11-04).
Every row ran on the test install graft-m1-site-a/b/c in the Milestone 0 account, or, where the table says so, in the crate's simulator or with a stub. Results are in tests/m1/results/, one file per work package, and each design fix they found went into the design first (decisions/DECISIONS.md, "Decisions from the M1 builds, 2026-11-01", "BM-1 and the M1-WP8 fixes, 2026-11-02", "The hosted closing runs, 2026-11-02", and "BM-1's 24-hour half and F43, 2026-11-02").
| Row | State | What closes it |
|---|---|---|
| BF-10 | Pass on Durable Objects: 1,000 of 1,000 chains with one group from two edges at once, and 100 of 100 during an owner move with the bootstrap claim (m1-wp7.md) | Done |
| BF-11 | Pass on Durable Objects: 1,500 of 1,500 cold-cache requests found the group after a member retry, a ReplaceMember stand-in, and an owner move (m1-wp7.md). ReplaceMember without its Workflow, spare, or Joint | Done for M1; the full ReplaceMember is M6-WP1 |
| BF-12 | Pass on Durable Objects: 10,000 of 10,000 kills kept their outcome on the register, and 1,000 of 1,000 on Push through the git edge (m1-wp2.md, m1-wp6.md) | Done |
| BF-13 | Pass on Durable Objects: 0 refusals from uncommitted entries in 400 rounds, plus 120 with early AppendEntries; Porcupine Ok (m1-wp2.md) | Done |
| BF-14 | Pass on the hosted edge: 2,400 commands across Push and identity (PutAccount) commands each missed its first receipt, half held 2.6 s and half dropped, and 0 were forwarded to a second member; also 0 in the simulator (m1-wp8.md, "Gate rows closed"; m1-wp3.md). The misses were injected by the test-faults build, not real network loss of a reply | Done |
| BF-15 | Pass on Durable Objects: 1,000 of 1,000 queued commands refused 401 after a revocation, and after a suspension (m1-wp7.md) | Done |
| BF-16 | Pass on the real groups (m1-wp6.md) | Done |
| BF-17 | Pass on the real groups, 12 of 12 rows (m1-wp6.md) | Done |
| BF-18 | Pass with a real ChainLog on a real repository group (run 2): git fsck clean on both shards, 9 of 9 clones matched, and the second shard served the newer pushes. Run 1 failed on a thin stored pack in the promoted range, now a design rule (Shape, "Packs"). Stubs: a test route triggers the archive, a key in one bucket is not re-put, last_promoted's index is an upper bound, consumed is empty, and RepoFold::check requires every archived range to be promoted (m1-wp5.md, "BF-18 on the hosted install") | Done |
| BF-19 | Pass on the hosted cron with the real request table: 0 of 1,000 committed packs deleted, 1,000 of 1,000 missing markers repaired, and 1,000 of 1,000 failed placements deleted, none before 24 h. Stub: the 24 h ran on the sweeper's injected clock, not a day of waiting (m1-wp4.md, "BF-19 on the hosted cron") | Done |
| BM-1, git half | Pass with site 0's member class failing every call for the rest of each run: clients.sh 29 of 29 on git 2.39.5 and 2.51.0; t/ on 2.51.0 and 2.56.0 lost no baseline pass except t5549 #2, which sets a server-side receive.hiderefs the host does not apply inference (m1-wp8.md). It also passes a gray fault, site 0's calls failing 30%: clients.sh 29 of 29 on 2.39.5 and 2.51.0, every leader move an election and no STEP_DOWN on any member (m1-wp8.md, "BM-1 gray-fault rerun"). Not covered: t/ under the gray fault, other rates or sites, and a no-fault t/ baseline on 2.39.5 (compared with 2.51.0 instead) | Done; the 2.39.5 baseline goes with BM-1's next rerun |
| BM-1, 24 h linearizability | Pass on lin24b (tests/m1/lin/raw/lin24b.report.json; m1-wp8.md, "The 24-hour half, 2026-11-04"; m1-wp2.md, "The 24-hour rerun's result, 2026-11-04"): repo:934783, 2026-11-03T00:55:07Z to 2026-11-04T00:55:07Z, 2,850,523 operations, 0 without a reply, p50 314 ms, p99 952 ms, under 4,176 resets (2,795 of a leader), 2,752 transfers (855 abandoned), and fault counters in 97 isolates. Porcupine Ok on all 6 keys widened 100 ms and 153 ms (the measured 305 ms ATL-SIN bound, halved); on the recorded clocks Illegal on k0 to k4 in 20 segments, 18 reads or pending writes of client rows lost in 15 of 12,535 cut chunks (Ok with them restored) and 2 cross-colo ties needing 1 ms and 4 ms of skew inside the measured bounds, none same-colo. 0 of 904,294 acknowledged writes lost, three identical logs of 513,232 entries, 0 of 805,201 refused CAS writes in the log, the member-side ack check 0 violations on 904,294 writes, and B0-11's same-clock early-ack test 0. That answers F44 as a client-clock artifact (decisions/DECISIONS.md, "F44 answered: the fast acks were a client-clock artifact"). Before it, lin24 did not pass: on repo:915635, 24 h to 2026-11-02T18:27Z: 2,739,691 operations, 0 without a reply, under 5,547 resets, 3,743 transfers (1,131 abandoned), and faults in 88 isolates. Porcupine Ok on all 6 keys widened 100 ms and 275 ms (the measured 550 ms LHR-SIN bound, halved); on the recorded clocks Illegal on k2, k4, and k5, all 5 segments reads of writes whose client rows were lost in 5 of 12,518 cut chunks, and Ok with them restored. 0 of 869,257 acknowledged writes lost, 0 of 774,418 refused CAS writes in the log, three identical logs of 479,965 entries, and B0-11's same-clock early-ack test 0. An early ack was not ruled out: entry 398032 needed 93 ms of ATL-LHR skew against a measured 49 ms, and 17 writes were acknowledged faster than any quorum could commit them on their own client's clock (m1-wp8.md, "The 24-hour half, 2026-11-02"; tests/m1/lin/raw/lin24.report.json). lin24b's 30-minute smoke had passed: 57,137 operations, Porcupine Ok on the recorded clocks and both widenings, and the ack check Ok on 18,225 of 18,225 writes (m1-wp8.md, "The 24-hour half, 2026-11-03"). Every history is checked per key with Porcupine's segmented mode, which now cuts each key on its own and keeps a no-reply write that a later operation observed; the old mode dropped every no-reply write, which can make a linearizable history look Illegal, and cut all keys of a file as one list. Rerun on the fixed checker, lin24's recorded-clock and 100 ms verdicts are unchanged; the 275 ms check needed over 7 GB for one key and was not rerun, and stands because its input has no no-reply writes and one key per file, the path both versions share (tests/m1/results/porcupine-seg.md). lin24b's finish builds the fixed checker. lin24b's pass rule was zero Porcupine violations, no acknowledged entry lost, and zero violations of the member-side ack check's six rules (tests/m1/results/m1-wp2.md, "Member-side ack evidence, 2026-11-03") | Done |
M1-WP2. Raft group core
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-12 | Leader killed after quorum and before the reply, 10,000 times. The edge re-sends with the same RequestId | Original outcomes returned, zero double applies | Any double apply or changed outcome |
| BF-13 | Leader lost before commit, on entries with a Stale refusal caused by an earlier command in the same entry | Zero refusals returned from uncommitted entries (Porcupine) | Any refusal returned from an uncommitted entry |
M1-WP3. Leadership and health
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-14 | Receipts that miss CALL_DEADLINE | Zero commands forwarded to a second member | Any command forwarded |
M1-WP4. Pack store and R2 keys
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-19 | 1,000 failed placements, plus 1,000 commits whose marker write is killed | Failed ones deleted after 24 h, zero committed ones deleted, missing markers repaired | Any committed pack deleted |
M1-WP5. Git cache shards and the git container image
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-18 | Archive a range of inline packs, kill all shards, rehydrate. A second shard behind its IngestCursor | git fsck clean, every ref clones, second shard serves the newer pushes | Any missing object |
M1-WP6. The edge for git over HTTPS and the clone cache
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-16 | Edge on N+1 with a new message format that is not enabled, plus one forced unknown variant | Nothing sent early. Unknown variant gives 503 with Retry-After and no state change | Any other rendering or any state change |
| BF-17 | Atomic push of 10,000 refs with a 1 MiB pack. A push whose updates exceed COMMAND_MAX | First: pack stored and push commits. Second: refused whole, no entry | Partial push, or an entry over 1.5 MB |
M1-WP7. Minimal identity and owner placement records
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-10 | 1,000 chain creations with two edges at once, plus 100 during an owner record change | One group per chain. Members without a committed config serve no read and grant no vote | Any chain with two groups, or any read or vote served without a config |
| BF-11 | Edges with cold caches after a member retry, ReplaceMember, and an owner move | 100% of requests find the group | Any request that finds no group |
| BF-15 | Revoke a stored token, and separately suspend an account, each with 1,000 of its commands queued | 100% of commands decided afterwards get 401 | Any accepted |
M1-WP8. Milestone 1 gate
Acceptance: the gate's pass and fail lines
| Row | Gate (Pass) |
|---|---|
| BM-1 | The git conformance suite and git clone / fetch / push --atomic from unmodified git 2.4x pass with one site's member class failing mid-run. Linearizability rerun at 24 hours. The git conformance suite is SHARED's git-http and git-protocol-v2 oracle rows on unmodified git 2.39 and 2.51, plus git's own smart-HTTP transport tests from t/ that can be pointed at a remote host, run by tests/m1/wp6/conformance/run.sh (decisions/DECISIONS.md, "Decisions from the M1 builds, 2026-11-01") |
build/BUILD-PLAN.md
Milestone 2: folds, issues, pull requests, and merge
Milestone 2 status (2026-11-07)
Milestone 2 is closed: every gate row below is Done, the last being BM-2 as reworded, on bm2-home (2026-11-06 to 2026-11-07).
Every hosted row ran on the test install graft-m1-site-a/b/c and graft-m1-edge, or on graft-m2b where the table says so (graft-m1 gained a fourth site script, graft-m1-site-d in wnam, on 2026-11-08 for BF-30, and graft-m2b gained graft-m2b-ops, Worker c53e4459 with the Secrets Store secret graft-m2b-mint-wrap-key, on 2026-11-09, when graft-m2b-edge became c4eb6915 with formats 32.8, 32.12, 32.14, and 18.3 added (AM1-WP6), then b9f4291e with the same formats after fanout's site day on 2026-11-10 (AM1-WP1); Milestone 2 ran before both), with real gh 2.102.1, git, and git-lfs 3.8.0 clients. LF-13 ran on a third test install, graft-m3c (tests/am1/results/install-m3c.md), built 2026-11-11 in account be2ae00c888cc5865939bb290fa07074: install 01M742HEA2SBCYP1VW26624KMW, sites graft-m3c-site-a (enam), -b (weur), and -c (apac), edge graft-m3c-edge with the agent formats and the tools edge graft-m3c-tools-edge, and no Secrets Store or graft-m3c-ops, because genesis hit maximum_stores_exceeded (the account's one store is graft-m2b's), so installation tokens do not mint there. Results are in tests/m2/results/, one file per work package, and each design fix they found went into the design first (decisions/DECISIONS.md, the groups from "BF-20 passes after the restore fix, 2026-11-04" to "BM-2 reworded by the operator, bm2-repack, and M3-WP4's pure half, 2026-11-06"). M2-WP6's final build rechecked the earlier rows at reduced counts, not as full reruns ("Per-row recheck on the final build", under M2-WP6).
| Row | State | What closes it |
|---|---|---|
| BF-20 | Pass at 300 commands/s on the 3.99 GB metadata tier of repo:196628: digest equal to a refold from both R2 copies and on all three members, apply p99 0 ms, no limit outcome, 423,000 of 423,000 commands (m2-wp1.md, "2026-11-04: restore fix and BF-20 rerun"). Rechecked on M2-WP6's final build only as a smoke: digests agree at two indexes after two site days, and 3 pushes; the 300/s campaign was not rerun | Done |
| BF-22 | Pass in the pure half at the row's 10,000 check runs and statuses on two members, prune included (M2-WP2). Hosted at 500 statuses (bf22e, m2-wp6.md, "BF-22 reduced"): 500 of 500 read through gh pr checks and REST, core rows unchanged but for consumer_cursor, both tiers' digests equal on every member, after two fixes. The prune clause ran only in the pure test, because the install's entry clock cannot be moved | Done for M2; a hosted prune needs a movable entry clock |
| BF-1 | Pass hosted, 3 of 3 (m2-wp3.md); rechecked 2 of 2 on M2-WP6's final build (bf1s) and on bm2-repack's | Done |
| BF-21 | Pass hosted, 100 runs with the leader killed in 10 (m2-wp3.md); rechecked 10 of 10 with 5 kills on M2-WP6's final build (bf21t), on bm2-latency's, and on bm2-repack's | Done |
| BF-23 | Pass hosted, 100 of 100 (m2-wp3.md); rechecked 20 of 20 on M2-WP6's builds, and 10 of 10 on bm2-latency's and on bm2-repack's | Done |
| BF-24 | Pass on graft-m2b, 75 of 75 within 10 s with the in-band first attempt, clean p99 6.44 s, conflict p99 5.81 s, with leader-change, reset, and purge runs (m2-wp5.md, "2026-11-05"); rechecked 20 of 20 on M2-WP6's builds (clean 0.83 to 2.10 s, conflicts 1.03 to 1.71 s) and once after the edge-wiring site day (6.3 s, m2-edge.md) | Done |
| BF-25 | Pass with real git-lfs and gh release: every verify and asset 201 commits only after the member confirms two buckets in two regions, with each blob's marker in all three buckets (m2-edge.md); rechecked as a smoke on M2-WP6's build. The unreadable-bucket clause ran only in the pure test, 60 of 60 asset and LFS reads with each bucket in turn unreadable, plus one corrupt copy (crates/api/tests/blobs.rs), fewer than the row's 100, because the install has no hook to make a bucket unreadable (m2-wp6.md, "BF-25 and BF-26 smoke") | Done for M2; a hosted unreadable-bucket run at 100 reads needs a bucket fault hook |
| BF-26 | Pass in the pure half with real git archive tar.gz and zip, and hosted: all 6 archive-cache/ objects deleted, then gh release download --archive=tar.gz gave the same bytes and rebuilt the object (m2-wp6.md, "BF-25 and BF-26 smoke") | Done |
| BM-2 | Pass as reworded on bm2-home: home clause p99 1.36 s at n = 300 with an EWR client and edge, 0 over 2 s; off-home clause p99 1.83 s at n = 288 from LAX, 0 over 3 s; every sample on a new term. Journeys 2 and 3 passed 10 of 10 on bm2-repack's build and stale parent_tip 100 of 100 on bm2f, neither rerun on bm2-home's (M2-WP6, "BM-2 status") | Done |
M2-WP1. Folds in member SQLite
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-20 | Snapshot of a 4 GB guess metadata tier under 300 commands/s | Apply p99 within 10% guess. Snapshot digest equals a refold at to. No request-limit errors | Apply p99 more than 10% slower, a digest mismatch, or any request-limit error |
M2-WP2. Issues, pull requests, reviews, and checks
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-22 | 10,000 non-required check runs and statuses on one repository, then entry time advanced past check_retention_days | gh pr checks and GET /commits/{sha}/statuses read them; core tier bytes unchanged; rows pruned on every member with equal fold_digest | A non-required row in the core tier, or members disagreeing |
M2-WP3. Merge and the merge queue
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-1 | One entry holding a 5-PR merge-queue group plus a PrSync batch | It commits on all three sites | Any follower refuses it |
| BF-21 | A group of 5 PRs under HeadGreen, the 3rd failing its required check on its queue ref, 100 runs, with the base leader killed mid-landing in 10 | The base moves only by QueueMerge entries that land a whole group; the failing PR is ejected and the other 4 land after a rebuild; every base move is a fast-forward | Any partial group, any non-fast-forward base move, or a queued PR lost |
| BF-23 | A required check passes, then is re-requested before the merge | The merge is refused until the new run concludes, 100 of 100 | A merge on the stale success |
M2-WP4. The wire layer, the blob store, and the other T0 repository surfaces
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-25 | git lfs push and gh release upload, then one site bucket made unreadable | verify and the 201 come only after two buckets hold the blob; fetch and download succeed 100 of 100 | Any acknowledged blob unreadable |
| BF-26 | All ArchiveCache objects deleted, then gh release download --archive=tar.gz | The bytes match the earlier download | Any difference or error |
M2-WP5. Cursors, Queues, and the PrSync consumer
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-24 | A head push, then git ls-remote origin refs/pull/N/merge; a conflicting push | The TestMerged commit within 10 s with a matching mergeable; on a conflict the ref is deleted and mergeable: false | A stale ref |
M2-WP6. Milestone 2 gate
Acceptance: the gate's pass and fail lines
| Row | Gate (Pass) |
|---|---|
| BM-2 | gh pr and gh issue journeys 2 and 3 pass, the merge p99 on a freshly elected leader on the group's home site, with the client in that region, is at most 2 s, and on a freshly elected leader off the home site at most 3 s, and a queue entry with a stale parent_tip is refused 100 times out of 100 |
| Clause | Result |
|---|---|
Journey 2 (gh pr review, required check, merge, view, list) | Pass, 10 of 10 (j2b): the merge is refused until the review and the check commit, then lands with parents (base, head), and gh pr view and gh pr list --state merged both show it |
Journey 3 (gh label, gh issue create -l, gh issue list, REST issues) | Pass, 10 of 10 (j3b) |
| Merge p99 on a freshly elected leader at most 2 s (old wording) | After bm2-repack: p99 2.42 s, p50 1.52 s, n = 288 on 300 new terms (final); per new leader site p99 enam 2.01 s (n = 66), weur 1.96 s (n = 120), apac 2.71 s (n = 102); 8 of 288 over 2 s. Before it, after bm2-latency: p99 4.10 s, p50 2.11 s, p90 2.93 s, n = 288 on 288 new terms (final); per new leader site p99 enam 2.40 s, weur 4.15 s, apac 2.89 s; 178 of 288 over 2 s, where a pass at this n allows 2. The first run, before the cuts: p99 3.20 s, n = 91 |
Stale parent_tip refused 100 of 100 | Pass, 100 of 100 on the final build (bm2f); not rerun on bm2-repack, which does not touch the queue's decide |
| Row | Recheck | Result |
|---|---|---|
| BF-1 | 2 queue runs (bf1s) | Pass 2 of 2: each landing's 5 Merged, Landed, and one 15-update Refs in one entry on all 3 members with one digest |
| BF-20 | Smoke: digests at 1720 and 1738 on repo:196628 after two site days, and 3 pushes | Smoke passes: all three members agree; the first pushes after two idle days took 7.1 and 14.4 s (cold tier load); the 300/s campaign was not rerun |
| BF-21 | 10 runs, 5 leader kills (bf21t, sd10) | Pass 10 of 10, each kill ending whole in 93 to 120 s; the first try (bf21r, 9 of 10) stalled 9.5 min on a stopped container, fixed by 576badf and 4c3727e |
| BF-22 | 500 statuses, 8 posters (bf22e, sd8) | Pass after two fixes: 500 of 500 through gh pr checks and 5 REST pages, core rows unchanged but for consumer_cursor, both tiers' digests equal on all members; the prune clause stays with the pure test |
| BF-23 | 10 + 10 (bf23r sd7, bf23s sd10) | Pass 20 of 20 (merge 8, squash 6, rebase 6) |
| BF-24 | 10 + 10, with conflicts and a transfer (sd7, sd10) | Pass 20 of 20 within 30 s: clean 0.83 to 2.10 s, conflicts 1.03 to 1.71 s |
| BF-25 | git lfs and gh release smoke | LFS and asset bytes match, both blobs in all 3 buckets; the unreadable-bucket clause has no hosted hook |
| BF-26 | All 6 archive-cache/ objects deleted, then gh release download --archive=tar.gz | Pass: same bytes, object rebuilt |
BM-2 stale parent_tip | 100 (bm2f, sd10) | Pass 100 of 100 |
build/BUILD-PLAN.md
Milestone 3: identity and authorization
Milestone 3 status (2026-11-09)
Milestone 3 is closed: every gate row below is Done, the last being BF-34's TestAccGithubRepositoryCollaborator on m3-repos (2026-11-09, tests/m3/results/m3-repos.md). BF-30 ran on graft-m1 (m3-finish, merged at b767933); the m3-rows rows ran on graft-m2b after four site days, the first onto main's build e2c67b5 and the last onto db1c6be (m3-rows.md, "What was deployed on graft-m2b"); m3-repos ran on graft-m2b after a site day from 3160687. The two test installs: graft-m1 has four site scripts since 2026-11-08, graft-m1-site-a to graft-m1-site-c and graft-m1-site-d in wnam (added for BF-30's move across placements), with graft-m1-edge, the tools edge graft-m1-wp7-edge, and graft-m1-ops; graft-m2b has three, graft-m2b-site-a to -c, with graft-m2b-edge and graft-m2b-ops. Three rows rest partly on runs outside the hosted install, and the table says which: BF-27's 1,000-queued count ran only in memory, BF-32's no-secret dump ran only in memory, and BM-3's learner half counts B0-18's graft-m2b run as a judgement.
| Row | State | What closes it |
|---|---|---|
| BF-2 | Pass on graft-m2b, 9 of 9 runs: 600 of 600 pushes held behind a revocation refused 401, 0 acknowledged at or after the revocation's index; the 3 controls acked 300 from reads below it (m3-rows.md, "BF-2") | Done |
| BF-31 | Pass on graft-m2b, 10 of 10: a scoped github_pat_ refused outside its scope; revoking either of a ghu_/ghr_ pair refuses the other; DELETE /applications/{client_id}/token revokes; 0 uses after revocation (m3-rows.md, "BF-31") | Done |
| BF-3 | Pass hosted on graft-m1: 200 of 200 pushes refused after a suspension, one key from two requests 100 of 100 with one owner (m3-close.md, "BF-3"); the SSH push is BF-52 (M6-WP2) | Done |
| BF-30 | Pass on graft-m1, 20 of 20 transfers and 0 of 2,313 route samples lost: across enterprises, across placements (Tri [0,1,2] to Na3 [0,3,1] home 3, with the new graft-m1-site-d in wnam), both at once, home only, and back; deps named the new owner after Settings in 20 of 20; a finisher killed after Reserved finished by Audit alone in 76.2 s with 0 of 586 samples lost; an org's enterprise change gave 2 of 2 repositories their Settings; 9 of 9 planted bugs (m3-finish.md, "BF-30") | Done |
| BF-33 | Pass on graft-m2b: with the new token's Tokens leader killed after UseCode committed and before the reply, 99 of 100 completed with exactly one token and 1 restarted cleanly; replay and codes older than 900 s refused 100 of 100 each, kept from an earlier run on the same install (m3-rows.md, "BF-33") | Done |
| BF-34 | Pass on graft-m2b: the driver 10 of 10 with 170 of 170 replies valid against GHES 3.22 (m3-rows.md, "BF-34"), and Terraform's TestAccGithubRepositoryCollaborator as written 6 of 6 runs, 12 of 12 subtests (terraform-provider-github 1b9c5c1, Terraform 1.17.1), every one of the 12 repositories it created answering 404 afterwards; its third subtest skips without GH_TEST_COLLABORATOR, as written (m3-repos.md, "Terraform") | Done |
| BF-35 | Pass on graft-m2b, 100 of 100 through gh ssh-key list and GET /user/keys with no Audit consumer (m3-rows.md, "BF-35") | Done |
| BF-36 | Pass on graft-m2b, 10 of 10: a real OIDC code flow with PKCE against a test IdP, a Password sign-in WrongMethod after the switch, a non-admin's write 403 (m3-rows.md, "BF-36") | Done |
| BF-27 | Pass in memory at the row's count, 1,000 of 1,000 refused (m3-wp4.md); hosted only through BM-3's 100 of 100 ghs_ refused after revoke, not with 1,000 queued | Done, on that evidence |
| BF-32 | Pass hosted, 12 of 12 twice with @octokit/app 16.1.4 (m3-hosted.md); the no-secret dump of every Identity chain ran in memory | Done |
| BF-28, BF-29 | Pass hosted on graft-m2b (M3-WP5 status) | Done |
| BM-3 | Pass, journey 4 100 rounds on edge 8a824825, with the learner half counting B0-18's 6 of 6 as a judgement (M3-WP6, "BM-3 status") | Done |
M3-WP2. Token kinds and short-lived codes
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-2 | Revoke a token while 100 of its pushes wait in admission | None is acknowledged at an index after the revocation's | Any one is |
| BF-31 | A github_pat_ scoped to one repository with contents: read; a ghu_ and ghr_ refresh; DELETE /applications/{client_id}/token | The PAT is refused outside its scope; revoking either paired token refuses the other; the DELETE revokes | Any use after revocation |
M3-WP3. Authz, Identity, and sign-in
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-3 | Suspend a user, then push with their PAT; separately, add one SSH key from two edges at once | Every push refused; the key gets one owner | Any push accepted, or two owners |
| BF-30 | Transfer across enterprises and placements, plus an org changing enterprise | Both routes reach the group throughout. deps names the new owner after Settings | Any lost route, or a stale owner after Settings |
| BF-33 | gh auth login device flow with the code shard's leader killed mid-flow, 100 runs; replay of a used code; a code older than 900 s | Each login completes or restarts cleanly; replay and the expired code fail 100 of 100 | A token from a reused or expired code |
| BF-34 | TestAccGithubRepositoryCollaborator; an org invitation accepted and one declined; a Collaborator write for an org member | Accepting grants in the same entry; declining grants nothing; the member write is refused | Any overlap or lost grant |
| BF-35 | A new SSH key and PAT, listed at once with no Audit consumer deployed | gh ssh-key list and GET /user/keys show them 100 of 100 | Any listing that misses them |
| BF-36 | A site admin switches InstanceAuth from Builtin to OIDC | The next sign-in uses OIDC; a non-admin write of it is refused | Any other result |
M3-WP4. Stateless installation tokens in graft-ops
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-27 | Revoke a jti with 1,000 commands under its installation token queued | 100% of commands decided afterwards get 401 | Any accepted |
| BF-32 | @octokit/app installation-token flow with a newly registered App; OAuth web flow with a registered OAuth app and a wrong secret | The JWT verifies against AppRecord.keys; the wrong secret gets 401; a dump of every Identity chain shows no client secret | Any other result |
M3-WP5. Learners and snapshot install
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-28 | A learner on its voter's site, and two learners on one site | Distinct objects | A name collision |
| BF-29 | Follower offline past the archive boundary, then rejoins | Catches up by InstallSnapshot with matching fold digests | Stuck or divergent |
M3-WP6. Milestone 3 gate
Acceptance: the gate's pass and fail lines
| Row | Gate (Pass) |
|---|---|
| BM-3 | Journey 4 passes over HTTPS, with revoked tokens, a revoked stateless installation token included, refused on the next write 100 times out of 100. Learners rerun the Milestone 0 learner gate |
build/BUILD-PLAN.md
AM1: base agent changes, leases, and tasks (built with M2 and M3)
AM1-WP2. Agent actors and refusals in decide
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| LF-2 | Unflag an AgentApp (DELETE /agents/apps/{id}), then push under its old confine prefix | The next decide after the acknowledgment no longer classes it ActorKind::Agent | A decision after the acknowledgment uses the old row |
AM1-WP3. External tasks and the task read endpoints
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| LF-1 | Uninstall an App with 100 open tasks across 10 repositories | Every task reads cancelled at the next read with no sweep command, and its next command records it | Any task reads live |
AM1-WP4. Leases, PathSet, and the activity feed
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| LF-3 | Feed cursor from another group, wait=40, 33 paths | 410 with oldest_cursor, wait clamped to 25, 422 for the 33 paths | Any other status, or wait above 25 |
| LF-5 | graftyard-plan blocks: a valid one, a repeated key, a bad glob, a second block, and a block over 4 KiB | One lease from the valid block. None from the others, and the body unchanged | A lease from an invalid block, no lease from the valid one, or a changed body |
AM1-WP5. The trust view
Acceptance: the gate's pass and fail lines
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| LF-4 | Trust view: one payload per HIDDEN_CHARS range in each graftyard_trust resource | 100% absent from visible and counted | Any present |
AM1-WP6. AM1 gate
Acceptance: the gate's pass and fail lines
| Row | Gate (Pass) |
|---|---|
| LM-1 | Base journeys pass unchanged, the base's agent rows rerun on real code, the base's 30-day hot-group storage row passes on a group seeded with the 4 GB tier, the leases and feed, lease limits, gate and read permissions, and core-tier rows pass, and the byte diff passes. Fleets of A1's size on one repository are enabled only after metadata child groups (base Milestone 7) |
| Day | Commit | Sites a / b / c / d | Edge | Ops | Notes |
|---|---|---|---|---|---|
| 2 | 40708cb | be79725f / aed638b3 / 232474cc / 150bab9c | 19e72fe6 | 1e9eb629 (the task claim) | New shard image sha256:5db19b4e…; --roll-shards stopped on a catch-up timeout and tests/am1/hosted/roll.mjs finished the roll |
| 3 | a4dd646 | c2f4c5e4 / bde58f9a / 6fe455bf / 69beff51 | 98de7bf4 | no roll | |
| 4 | f925d40 | ed0bff49 / 8686876c / 11ff63a5 / f47c2827 | b330c05c | no roll | |
| 5 | c836d14 | b3d42554 / 7bed34dc / e57c7482 / 0b7bcdf4 | d3a8981f | no roll; L0-16's two fixes | |
| 6 | 04e4f22 | 6553b6dc / 751b0701 / 98221437 / 002551dc | ce2e1ac3 | no roll; main through 5d18dda merged, with the read gate | |
| 7 | c7384ec | b5bfd297 / a0ee9724 / da589789 / 9199c4e4 | 382356ed | unchanged | no roll; am1-fix's two fixes (tests/am1/results/am1-fix.md) |
| Row | Hosted result |
|---|---|
| Base journeys | Pass: j2 and j3 20 of 20 each; journey 4 PATs and ghs_ 10 of 10 refused after revoke |
| L0-11 (byte diff) | Pass: day 2 0 of 2,048 replies differ; the day 5 and day 6 differences are only j3's new issues and the read gate's anonymous 404s |
| L0-12, refusals | Pass for all six kinds (am1-runs): the confined vendor App 100 of 100 pushes and 100 of 100 REST ref writes refused naming the lease; the Graftyard-run App (graftyard-workspace) 100 of 100 of each refused, but by NoLiveTask ("push needs a live task") before the lease check, because the edge cannot yet create a Graftyard-run task (AM3-WP3); 0 of 100 for a flagged App without confine, an unflagged App, a human PAT, and Graftyard MCP |
| L0-12, stall | **Pass after am1-fix** (day 7, lf1-5): a 30-minute stall with every member call failing, then a leader change; term 3's first entry carried extend { by: 1,761,355 ms }, all 20 leases moved by 1,761 or 1,762 s, exactly one entry carried extend, and clock spread was at most 65 ms. Before the fix (day 5) a new leader wrote no extend and leases moved 0 ms. The fix (96dc541): a follower marks the stall from the leader's late re-send of an uncommitted entry; the long sweep passed (335.75 s) and Porcupine was Ok on both histories. Uncovered: an old leader that never reaches the winner, which loses the stall as a lost marker does |
| L0-12, A1 replay | Fail (am1-runs, 500 tasks and 500 Exclusive leases, a leader kill every 60 s): 0 events reached the 500 pollers in 600 s, of 149,265 polls (74,432 with no status, 52,602 gone, 12,099 notServing, 6,277 lagging, 3,855 held); 57 of 2,491 agent pushes ok; the kills did not land (1 of 10 kill pushes ok, term unchanged); 5 of 1,451 lease renewals answered 200 |
| L0-13 | Pass: 6 bad acquires 100 of 100 (am1-hosted); then (am1-runs) 600 + 600 plan blocks gave 0 plan leases from non-writers and 500 from writers (63 holders, at most 8 each); a holder removed from the org refused nothing at once (2 of 2 pushes refused before, 2 of 2 accepted after); the 5 h renew read shared at every read after 240 min |
| L0-15, feed half | Pass with the row's 404 (contradiction 1, now in the row); the AgentGate, Code Mode, /log, and prompt halves need MCP and the controller |
| L0-16 | Pass: 0 metadata reads on 3 members for 8 checks; a merge waits for the tier by design (the row no longer lists merges) |
| LF-1, LF-2, LF-3, LF-5 | Pass |
| LF-12 | **Pass after am1-fix** (day 7, lf1-6): leader commit p99 474 ms after place_learners (6,481 ms before the fix) and 710 ms before it, with 5,000 of 5,000 polls held on 11 learners and 0 on voters. The cause was a member call: each poll on a stale learner sent its own owner-Authz read to that chain's leader, which delayed the repository leader's dep read; the fix (760f9c0) keeps one owner read in flight per member. Through the edge from the orb, client p99 is about 2.7 s even with no polls: that is the client path, not the commit, which the row measures at the leader |
| LF-13 | **Pass on graft-m3c** (lf13-m3c, site day 4, 9d59283; tests/am1/results/lf13-m3c.md), both modes, 3 of 3 runs each, 999 polls held. Gap: a real gap in 3 of 3; 999 of 999 held polls returned with their cursor and no events within 1 s of the lift (99, 306, 799 ms) and of F's stop (0 ms); 1 of 50 probes lagging per run, because a gap ends at F's next accepted round; 0 events missing. Install: InstallSnapshot in 3 of 3; 999 of 999 within 1 s of the lift (788, 83, 437 ms) and of the stop (0 ms); probes lagging 48, 50, and 49 of 50; 0 missing, and the pre-fault cursor answered gone. Between the lift and F's stop, F admitted 0 to 2 of 50 probes, which its stop then answered with nothing skipped. Fixes: 3597c6f (a stop counter, so a stop between two waits still answers a held poll; and the test-faults /sever route), 429eede (a lease acquired and released in one catch-up batch reports both), and 7972504, 606b403, 9d59283 (the ring restarts at the install's base at once, with the installed tier's live leases); 7 of 7 planted bugs caught. Cost $2.55. Earlier, on graft-m2b (am1-runs to lm1-rerun2): no gap formed, and held polls ran out their own 25 s |
| LF-4 | Pass, all eight resources (lf4-rest, graft-m3c; tests/am1/results/lf4-rest.md): issues, pull requests, issue comments, reviews, review comments, commits, check runs, and releases 16 of 16 each, every hidden range absent from visible, invisible_chars 2, raw text unchanged, the untrusted writer's objects trusted: false, no graftyard_trust without the media type. The six new reads are REST GETs on the edge (F61), validated against the pinned description locally (96 REST replies) and hosted (16), 0 failed; 16 of 16 planted bugs caught. Rerun on lf4-rest-2 (tests/am1/results/lf4-rest-2.md) with review comments and check runs written through the new REST POSTs instead of a test-only commit: 8 of 8 resources 16 of 16 again, every write 201. Rerun on lf4-rest-3 (tests/am1/results/lf4-rest-3.md) with the check-run create's id taken from the IdBlock cursor in decide (repository 32.18): 16 concurrent creates 16 × 201 with 16 distinct ids from one block, 8 of 8 resources 16 of 16. Earlier (am1-runs): issues and releases only |
| B0-29 | Fail (am1-runs, 500 agents): commit p50/p99 1,205/9,316 ms (n 321); 5 of 2,500 agent pushes ok (1,772 × 503, 722 client timeouts); 1 of 99 merges; leader moved 0 → 2 with 2,305 failed member calls; 0 feed polls reached the leader |
| B0-31 | Fail: 4 of 2,501 agent pushes ok (1,974 × 503, 522 timeouts); the apac leader did not hold (back to site 0 in setup, ending on site 1, term 30), so no commit sample on the final leader, and the placement-failing half was not reached; feed p99 42.3 s |
| B0-32 | Fail: 0 of 500 burst pushes within 5 s, 346 × HTTP 503 and 154 timeouts, neither in the ng shape; 0 leader changes; commit p50/p99 714/1,199 ms (n 84); 72 of 755 agent pushes ok |
| B0-33 | Fail (5,000 agents, 300 s): about 0.6 agent commands/s admitted against 140; 9 of 12,454 agent pushes ok (12,194 × 503); the fast user 6 of 508 × 200 and 502 × 503; humans 4 of 487; commit p99 1,086 ms (n 53); 0 leader changes; 3,195 failed member calls |
| B0-34 | Fail: clone time to first byte p50/p99 23.1/86.7 s (80 × 200, 255 × 503); human pushes 0 of 33 ok; agent pushes 0 of 2,500 ok; commit p50/p99 5,516/5,659 ms; 0 leader changes; k stayed 1, with no fanout |
| B0-35 | Fail, not a valid run of its line: setup opened 10 of 500 PRs; mergeable polls 187 MERGEABLE and 9,386 × 503; 63 of 2,500 agent pushes ok; commit p99 13,213 ms; dispatch not measurable (no Actions on graft-m2b) |
| B0-30 | Fail on its fail line (tests/am1/results/b0-30.md; graft-m1, branch b0-30, merged into main; pushes off from simulated day 1.34). Member SQLite crossed CHILD_GROUP_AT (8,589,934,592 B) at 23:09:41Z on 2026-11-11, simulated day 27.52: site 0 read 8,591,708,160 B, and sites 1 and 2 crossed within 30 s. No child group took the tier, because MetaSplit and RepoMeta are types only (G6). The rest held: SQLITE_FULL 0 on every member throughout, and the final verify read 81 of 81 sampled archived ranges from both buckets and resolved 10 of 10 RestoredTo lookups. Rates: while archiving kept up (simulated days 7.08 to 22.27), SQLite grew 0.0585 GB a simulated day, which projects to about 8.21 GB at day 30, under the line inference, linear; during the stall it grew 0.184 GB a day. Cause of the stall: from about 17:13Z fanout (k = 6, with no entry found that raised it; A31) forwarded the archive promotion's /validate to cold pool shards, each of which read the multi-GB log tail before validating, so archive runs exceeded RUN_FOR (120 s); site a's promotion validate median went from 0.17 s at 16Z to 7,551 s at 21Z, and site b archived its last range at 20:17:56Z (G7). Blocker 12's cursor skip came back on site c, and the group was left with no leader three more times (F67). Cost $64.90. The earlier substitution is withdrawn ("What LM-1 still needs") |
| B0-36 | Pass (am1-runs) on every surface the deployed edge routes, 100 per action, for the confined vendor App (m3-ci) and the Graftyard-run App (graftyard-agent) with its shipped permissions and on an installation widened to every permission; ready-for-review and REST issues/{n}/labels included. Graftyard MCP, the control, met no agent refusal. The Graftyard-run kind's git-surface refusals came from NoLiveTask; its lease refusal and task pushes moved to AM3-WP3 (operator, 2026-11-10). GraphQL addLabelsToLabelable, rerun on graft-m3c (lf4-rest): 100 of 100 refused by the allowlist rule for m3-ci and for graftyard-agent shipped and widened, 0 of 100 for Graftyard MCP, a label inside the allowlist accepted 5 of 5 for every kind; the same again on lf4-rest-2, with m3-ci flagged through PUT /api/graftyard/v1/agents/apps/{id} by the m3 admin, now a site admin of graft-m3c |
| B0-37 | Push half pass at 100 per kind (about 1,000 pushes, not 10,000): 0 parse failures, every forged trailer Claimed, 100 of 100 refs/graftyard/ writes refused. Fetch half partial: session refs stay hidden from v0, v2, and clone --mirror, but no session ref exists to list, because only graftyard-workspace under a live Graftyard-run task writes one (AM3-WP3); containers/git/server/sessions_test.go covers the listing |
| Row | Rerun on `graft-m2b` |
|---|---|
| B0-29 | Fail: 105 of 2,500 agent pushes ok (1,897 × 503 from the gate, 440 timeouts); commit p99 30,092 ms (n 9); merge p99 120 s; leader 0 → 2; 16,777 leader /reads and 20,199 /suspect in 600 s |
| B0-31 | Fail: 133 of 2,500 ok (1,777 × 503); commit p99 2,508 ms (n 12); the apac leader did not hold, and the placement-failing half was not reached |
| B0-32 | Fail: 0 of 500 burst pushes within 5 s (323 × 503, 132 × 500 "no member leads", 43 timeouts), none in the ng shape; leader 0 → 2 |
| B0-33 | Fail (5,000 agents): about 2.5 agent commands/s admitted against 140; 62 to 331 of 12,546 agent pushes ok (a driver parser bug, 994266c, bounds the count); commit p99 3,415 ms; no leader change |
| B0-34 | Fail: k stayed 1 (a follower's /fanout error); clone time to first byte p99 172.4 s; 0 of 37 human pushes ok; commit p99 361 ms |
| B0-35 | Fail: the setup now opens 500 of 500 PRs (am1-runs' 10 of 500 was very likely the same parser bug); mergeable polls 10 MERGEABLE and 10,162 × 503; 0 of 2,500 agent pushes ok |
| L0-12, A1 replay | Fail: 0 events reached the 500 pollers in 600 s of 211,151 polls; 0 of 10 kill pushes landed |
| LF-13 | Fail: install mode reached its fault in 4 of 6 runs, refused new polls (45 to 47 of 50 lagging), and lost no event (the ring answers gone, dfa5410 deployed), but F stopped its held polls in only 2 of 4; gap mode reached a real gap in 1 of 6, with 0 lagging and one lease event fewer on F's ring than on the third voter, an open bug candidate. The row's "within 1 s" does not say of what: the lift or F's stop |
| Row | `lm1-rerun` on `graft-m2b` |
|---|---|
| B0-29 | Fail: 274 of 2,500 agent pushes ok (1,425 × 503 from the gate, 224 × 500, 567 timeouts); commit p50/p99 1,063/1,505 ms (n 144); merges 2 of 3, p99 76 s; API reads 4,143 × 200, 9,576 × 503; leader held; exceededMemory 11,117 / 4,478 / 2,814 on sites a/b/c. With API reads off (diagnostic, 300 s): 600 of 1,249 pushes ok and clean for 2 minutes, then members ran out of memory |
| B0-31 | Fail: 48 of 2,500 ok (1,741 × 503); commit p99 2,359 ms (n 93); the placement-failing half was not reached |
| B0-32 | Fail: 0 of 500 burst pushes within 5 s (456 × 503, 44 timeouts), none in the ng shape; leader 0 → 2; 40,180 leader /reads and 51,362 /suspect |
| B0-33 | Fail (5,000 agents): 101 of 12,484 agent pushes ok (10,633 × 503); about 2.1 side commands/s admitted against 140; 0 of 29 human pushes; leader 0 → 1; members 331 / 262 / 250 MiB |
| B0-34 | Invalid: stopped at about 07:15Z on a false cost reading (account-wide rows that were B0-30's); 103 of 2,577 receive-packs answered 200; k stayed 1, though /fanout reports now answer ok (16 of 17 in B0-29) |
| B0-35 | Not run: it would have started at $11.42 against the $12 row stop |
| L0-12, A1 replay | Fail: 0 events timed at the pollers of 146,705 polls; 1 of 10 kill pushes landed; 137 of 2,490 agent pushes ok; members 283 / 280 / 265 MiB |
| LF-13 | Fail, with HELD=999. Install reached its fault in 2 of 3 runs: every held poll came back with its cursor and no events, 48 and 47 of 50 new polls answered lagging, and no event was lost (F answered gone), but only 630 and 479 of 999 returned within 1 s of the lift (the slowest at 1,003 and 1,004 ms), and F itself stopped only 193 and 0 of them; the rest returned before the lift at their own 25 s wait. Gap left no gap in 3 of 3 (O level with F each time) (A26) |
| Row | `lm1-rerun2` on `graft-m2b` |
|---|---|
Smoke, 250 agents, API reads off (ccd10b1 only) | 491 of 624 pushes ok; leader 0 → 2; members 280 / 113 / 105 MiB; 607 exceededMemory on site c |
| Smoke, 500 agents, API reads off | ccd10b1 only: 300 of 1,232 ok, 6,924 exceededMemory on site c. With eb8eb0b: 812 of 1,254 ok, still a leader change and 1,621 exceededMemory on site b |
| B0-29, twice | Fail. With eb8eb0b: 118 of 2,501 agent pushes ok (1,690 × 503 from the gate), commit p99 37,954 ms, members 273 / 247 / 201 MiB. With a271fb3: 107 of 2,501 ok (1,770 × 503), commit p50/p99 1,757/3,967 ms, members 267 / 265 / 263 MiB though a whole-tier read was about 150 KB; 87,000 leader /reads and 108,000 /suspect |
| B0-35 | Fail: the setup opened 500 of 500 PRs, then from about 17:08:30Z every edge request answered 503. The group was left with no leader: at 17:21Z, after the load stopped, all three voters were followers at term 4 with no leader, after 150,399 /suspect reports and 78,842 /reads to one of them, and one test-only /campaign to each elected site 0 within 5 s. Member peak 279 MiB. Diagnosed by lm1-memory: the term-2 leader f0 was stuck in one alarm invocation for 641 s wall (622 s CPU, 41.9 M rows read), building dumps for about 730 reads whose callers had gone (fixed in f125a37), while f1 kept hitting exceededMemory; elections were not at fault and crates/raft is unchanged. Test-merge vCPU and list-field computes are still not instrumented (AM1-WP1, "Follow-up: the test-merge budget and B0-35's instruments") |
| B0-32 | Fail: 12 of 500 burst pushes ok, none in the ng shape, p50 27 s; leader 0 → 2; 63,043 /suspect on the leader; member peak 318 MiB |
| B0-31 | Fail: 126 of 2,501 ok; commit p99 9,754 ms; the term churned from 3 to 28 in the run, with one voter at a time not answering /status; 128,734 /suspect on the leader; member peak 265 MiB |
| B0-33, B0-34, L0-12 A1 replay | Not run: the round's $13.40 no-new-row threshold was reached after B0-31 |
| LF-13 | Fail on graft-m2b (gap left no gap in 3 of 3; install's held polls returned up to 1,228 and 1,741 ms after the lift), its last run there. LF-13 has moved to its own install, graft-m3c (thread T-01a1c666-2b83-7522-8afd-45a508a3cd74) |
| Row | `lm1-memory` on `graft-m2b`, `d9ee594` |
|---|---|
| Smoke, 500 agents, 300 s | Every member under 80 MB (peaks 69 / 35 / 78 MB); 1,237 of 1,242 agent pushes ok (5 refused); commit p50/p99 793/2,956 ms at the leader; leader 0 → 2, on Confirm then CommitErrors |
| B0-29 | Fail: commit p99 3,867 ms against 600 ms (p50 1,000, n 2,077); leader 0 → 1; 1,089 of 2,500 agent pushes ok (666 stale, 406 timeouts, 205 missed); the apac member at 262 MB |
| B0-32 | Fail: a leader change, 71 suspect, commit p99 4,834 ms; 48 of the burst ok, 452 refused |
| B0-31 | Stopped early, before its row line |
| B0-33, B0-34, B0-35, L0-12 A1 replay | Not run: the 500-agent bar did not hold at 600 s, and the rows would have passed the cost cap with nothing new to test |
| Row | Gate (pass / fail) | Round 3 on `graft-m2b`, `54c45213` |
|---|---|---|
| Smoke, 500 agents, 300 s | None | Peaks 75 / 55 / 112 MB, 0 exceededMemory; the leader stepped down on CommitErrors 15 s into load; commit p99 5,675 ms |
| B0-29 | Commit p99 ≤ 600 ms, merge p99 ≤ 2 s, zero Stale on distinct refs, leader < 600 req/s with zero feed polls / commit p99 > 1 s, a queue merge > 10 s, a feed poll on the leader | Fail: commit p99 3,419 ms (p50 662, n 8,167); leader 0 → 1 → 2, the first step-down 7 minutes into load; 7,774 /suspect; 2,412 of 2,500 agent pushes ok; merge p99 9,303 ms; 306 req/s and 0 feed polls on the leader |
| B0-32 | Every push acked within 5 s or ng … admission full, merges in the next 2 entries, zero leader changes and suspect | Fail: zero leader changes but 3 suspect; burst 168 ok and 332 refused, not in the admission full shape, 1 within 5 s; merge p99 41,005 ms |
| B0-31 | Commit p99 ≤ 1 s with buckets up / > 3 s | Fail, between the lines: commit p99 2,541 ms; leader 0 → 1 in two moves (CommitErrors, then Confirm p99 1,377 ms) |
| B0-34 | k in band ≤ 8, human push p99 ≤ 3 s beyond transfer, clone TTFB p99 ≤ 1 s / TTFB p99 > 5 s, k > 8 | Fail: clone TTFB p99 58,554 ms (p50 1,643); k 1 → 2 in band; human push p99 240 s |
| B0-33, 5,000 agents | Human and merge-queue p99 ≤ 1 s, ≥ 140 agent commands/s, zero leader changes and suspect / a leader change or suspect, < 100 commands/s | Fail: term 2 → 19 in 5 minutes, 27,703 /suspect; commit p99 15,980 ms; 127 of 12,523 agent pushes ok, 6,925 answered 503 by the read gate |
| L0-12, A1 replay with a leader kill every 60 s | Feed p99 ≤ 2 s outside kills, no lease overlap over 500 ms, refusal counts right | Fail: feed p99 64,031 ms (p50 2,647); f0 led through 10 kills (term 2 → 5); f0 peaked at 275 MB with 711 aborted /reads |
| B0-35, 500 open PRs | Test-merge vCPU within TEST_MERGE_BUDGET, dispatch p99 ≤ 10 s, zero list-field computes | Not judged: the budget was not instrumented and graft-m2b has no Actions; mergeable 3,876 MERGEABLE and 24,168 × 503; f1 logged 812 exceededMemory |
| Row | Round 4 on `graft-m2b`, `48f8fae1` |
|---|---|
| Smoke, 500 agents, 300 s, 3 learners | Leader held term 2 with 0 /suspect; commit p99 3,966 ms; 1,218 of 1,241 agent pushes ok; read:api 15,913 × 200 and 36,422 × 503 with Retry-After: 1; leader peak 95 MB, no exceededMemory |
| B0-29 | Fail: commit p99 5,461 ms (p50 834, n 6,793) against 600 ms; 0 leader changes and 0 /suspect; 2,250 of 2,500 agent pushes ok; merges 13 of 28 answered, p99 18,971 ms; read:api 51,594 × 200 and 52,858 × 503; the org's Authz leader took 87 /read/s and stepped down |
| B0-32 | Fail: 0 leader changes and 0 /suspect, but the burst had 49 ok and 446 refused, not in the admission full shape, 0 within 5 s (p50 26.6 s) |
| B0-31 | Fail, between the lines: commit p99 2,311 ms (p50 658) against 1 s, under the 3 s fail line; 29 /suspect |
| L0-12, A1 replay with a leader kill every 60 s | Fail: feed p99 12,984 ms (p50 1,141) against 2 s; term 11 after 10 kills; voters peaked at 256 to 278 MB; lease overlap not reported by the driver |
| B0-34 | Fail: clone TTFB p99 158,542 ms (p50 846) against 1 s; k 1 → 2 in band; leader held term 2 with 0 /suspect; human push p99 121 s |
| B0-33, 5,000 agents | Fail: term 2 → 8, 7,539 /suspect; commit p99 8,568 ms; 2,020 of 12,533 agent pushes ok |
| B0-35, 500 open PRs | Not judged: the budget's instruments (merge-budget) were not in this build and graft-m2b has no Actions; 18,870 mergeable reads answered 503 |
| Row | Round 5 on `graft-m2b`, sites `7583378f`, edge `afb025ac` |
|---|---|
| B0-29 | Not judged on commit: the leader held (site 0, term 2, 0 changes, 0 feed polls), but its /stats split was lost when the isolate reset. Merges fail: 4 × 200 and 25 × 503. 2,305 of 2,500 agent pushes ok |
| B0-32 | Fail: burst 147 ok and 352 refused, not in the admission full shape (p50 23,063 ms); commit p99 932 ms; two CommitErrors holds, term 2 → 4 |
| B0-31 | Between the lines: commit p99 1,163 ms against 1 s, under the 3 s fail line (round 4: 2,311 ms); the leader held |
| L0-12, A1 replay with a leader kill every 60 s | Fail: feed p99 14,291 ms against 2 s; leader 0 → 1, term 6; the last leadership's Authz dep reads p50 2,246 ms and 956 /suspect. Lease overlap is not measurable: the driver takes only Shared leases, which never conflict |
| B0-34 | Fail on human push p99, 121 s (client timeouts). Clone TTFB p99 2,253 ms is between the lines (round 4: 158,542 ms); k 1 → 2 in band; the leader held |
| B0-35 | Judged, and fails over budget: test merges used 0.274 vCPU against 0.25 (707 computed, 6,043 shed). List-field computes 0, which passes. Dispatch p99 not instrumented (no Actions). mergeable 1,949 MERGEABLE, 16,018 UNKNOWN, and 39,990 × 503 |
| B0-33, 5,000 agents | Not run: the round's $10 stop came first |
build/BUILD-PLAN.md
Gates not yet run
Later milestones' pass and fail lines, fixed before the code is written. Nothing below has a result yet.
Milestone 4: events, webhooks, and Actions
M4-WP1. Webhooks and the remaining consumers
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-57 | One hook timing out at 10 s beside 49 healthy hooks of the same owner, 10 events/s | Healthy hooks' first-attempt p99 at most 2 s; the slow hook gets its events in chain order; Webhooks cursor lag at most 5 s | Healthy p99 over 10 s, an out-of-order delivery to one hook, or cursor lag that grows with the slow hook |
| BF-58 | Kill the site holding a hook owner's HookSequencer with deliveries pending | Every event in the range reaches every hook at least once, with unchanged GUIDs | Any event never delivered, or a redelivery under a new GUID |
M4-WP2. User records, the watcher index, the Inbox, and search
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-5 | Star a repository and comment on a watched issue, then delete and rebuild the watcher's Inbox | stargazers_count and /notifications show both within 10 s, and the rebuilt Inbox keeps read state | Lag over 60 s, or read state lost |
| BF-67 | Delete an Inbox and rebuild it. Also 10,000 stars, then a refold | Read and done state kept, nothing from sources lost. /stargazers lists all 10,000 | Any lost read or done state, a lost event from a sources chain, or a missing star |
| BF-68 | EU-resident org's search documents. Kill one SearchShard | Documents only in the eu-jurisdiction shards. incomplete_results until the rebuild, then full results | A document outside EU, a partial result without incomplete_results, or results wrong after the rebuild |
M4-WP3. The Actions engine
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-4 | Kill a runner mid-job, then leave its Actions group idle | The job fails within 10 minutes of its last renewjob | It is still running at 11 minutes |
| BF-37 | on: merge_group workflow, and go-playground/webhooks parsing merge_group, pull_request.enqueued, and pull_request.dequeued | A run on each built queue ref; every queue ref deleted within 5 min of landing or ejection guess bound | A missing delivery, or a queue ref left over |
| BF-54 | Hold 2,500 runner polls on one broker scope (BROKER_HELD_MAX plus 500) with jobs queued | Polls past the cap get 503; those runners re-poll within 30 s with their sessions valid; job start p99 at most 35 s for them | Any RunnerSessionInvalid, a runner that exits, or a job unstarted after 60 s |
| BF-55 | Kill the leader site of an Owner scope's registration shard during a 256-job matrix | Polls reach a broker on the new leader site within 15 s; every job runs once; zero jobs assigned twice | A job assigned twice, or dispatch stalled over 60 s |
| BF-56 | Kill the repository's commit path (and, separately, RunOwner) between RunCreated and the RequiredCheck writes, then finish or cancel every job | Every RequiredCheck on an open PR head has a conclusion within 60 s of its job's Completed | Any RequiredCheck with conclusion: None after its job completed |
| BF-59 | gh secret set for a repository, an environment, and an org, then a job reading all three; then rotate the SealKey and the wrapping key | The job sees all three values before and after both rotations, and a fork-PR job sees none | A wrong value, a value shown to a fork PR, or a plaintext secret found in any entry or log |
M4-WP4. Hosted runners on Sandboxes
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-60 | 50 ubuntu-latest jobs queued for one owner with the cap at 20, while two graft-runners sweeps overlap | At most 20 hosted runners at once; every job finishes; no Sandbox is left 10 minutes after the last job | More than 20 at once, a job that never finishes, or a leaked Sandbox |
M4-WP5. The web app inference placement
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-38 | Web app ("Frontend and native app" in design/SHARED.md) under Playwright inference: sign in, open a repository, a pull request's diff, checks, and timeline, and a running job's live log (the actions-logs-results oracle), then force the checks region's query to fail | Every page renders, the first route's data is inlined in the HTML shell with zero API requests before first paint, the log streams live, and with the checks query failing only the checks region shows an error | Any page fails, the first route waits on a second round trip, or the checks failure breaks another region |
| BF-61 | View a running job's live log in the web app, then kill its LogRelay | Lines appear within 2 s; after the kill, the viewer falls back to step log blobs and the final log is complete | A line later than 2 s, no fallback after the kill, or missing lines in the final log |
M4-WP6. Milestone 4 gate
| Row | Gate (Pass) |
|---|---|
| BM-4 | An unmodified actions/runner runs a matrix, and a site is failed mid-run with no lost job state |
Milestone 5: release control and site days
M5-WP1. The Release group, genesis, and keys
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-6 | Replay every historical Release entry, pre-signed undos and McpPointer and ControllerPointer entries included, after Keys { vendor: None } | 100 of 100 refused | Any accepted |
| BF-40 | Submit a second intent while one is current, and an intent naming a release with no ReleaseUploaded | Both refused | Either committed |
| BF-41 | Two genesis entries sent to an empty Release group: one matching the anchor, one signed by another key | Only the matching one commits | The other commits |
| BF-42 | Sign an entry offline while graft-ops commits an IntentStep | 409 with the current digest; the re-signed entry commits | A stale prev commits |
| BF-45 | A forged ReleaseUploaded, ReplaceMember, or pointer signed by the executor key, for a site not drained 24 h | Only a correctly signed ReleaseUploaded commits | A pin, or a ReplaceMember of an undrained site |
M5-WP2. graft-front, pin-only pointers, and gradual edge deploys
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-8 | Blackhole every Release member for 10 minutes, then set EdgePointer to a version outside the deployment | Requests keep the last pointers; the mismatch alarms within POINTER_TTL and the EdgeRoll that dropped it was refused | Any request blocked on the group, or a silent mismatch |
| BF-43 | EdgePointer undo under 1% EdgeRoll with the API blocked, then the next EdgeRoll | Every request pins to N within POINTER_TTL; the next EdgeRoll sets None | Any request off N while pinned, or the pin survives |
| BF-48 | Blackhole the Release group, then send a customer-signed StaleMode | Scoped groups serve stale() within POINTER_TTL; the flag expires at until | No stale reads, or a flag past STALE_MAX honored |
M5-WP3. The graft-ops executor and record formats
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-7 | After FormatsEnable, submit the pre-signed Rollback to N and Abort { intent, target: N } | Both refused, and both page | Either applied |
M5-WP4. Site days and automatic drain
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-39 | Kill graft-ops at each step of a SiteDay and an EdgeRoll, 100 times | It resumes after the last IntentStep; no step skipped; one IntentDone | A skipped or doubled step |
M5-WP5. The change feed, the export endpoints, and the vault
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-47 | release roll end to end, including SiteDay { Workflows } and graftyard vault upgrade | Order as in Release roll; vault upgraded with no install token | Any other order, or an install token used |
| BF-49 | Vault pulls every export endpoint; then the export jti is revoked | All four endpoints served; refused within READ_CACHE after revocation | Any served after READ_CACHE |
M5-WP6. Revocation and rotation runbooks
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-44 | A vendor-signed undo not in the parent's undo list, and a Resume after Keys { vendor: None } without customer re-signed undos | Both refused | Either accepted |
| BF-46 | Rotate of each SecretName with live traffic | No new script version; zero failed calls; old value unused after the switch | A new version, or failures |
M5-WP7. Milestone 5 gate
| Row | Gate (Pass) |
|---|---|
| BM-5 | A deliberately unreadable format released to graft-site-a commits nothing and moves A's groups. With the vendor token revoked mid-roll, Halt, Keys { vendor: None }, then Abort { intent, target: N } returns every site and pinned script to N, and a vendor-signed intent submitted after Keys { vendor: None } is refused. A bad graft-ops at 1% swaps back by OpsPointer with the API blocked |
AM2: graft-mcp (after M5)
AM2-WP1. graftyard apps init and the graft-mcp authorization server
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| LF-6 | token/scoped from Graftyard MCP without graftyard_grant, with another user's grant, and with graftyard_write above the grant. A stale generation on POST /mcp/grants/{id}/generation. Grant calls with wrong client credentials | 422 for each bad token request. 409 and the grant revoked for the stale generation. 401 for wrong credentials, 100 of 100 | Any accepted |
AM2-WP4. AM2 gate
| Row | Gate (Pass) |
|---|---|
| LM-2 | MCP row passes, and a bad graft-mcp at 1% swaps back with the API blocked |
AM3: the controller and Graftyard-run agents (after M5)
AM3-WP1. The agents account, graft-agents-front, and the controller
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| LF-7 | Bad graft-controller at 1% with the deployments API blocked. Submit the pre-signed ControllerPointer undo, then blackhole GET release/entries for 10 minutes | Every graft-agents-front isolate pins N within POINTER_TTL of its next request. Webhooks keep flowing on the last view | Any webhook reaches N+1 after the pin, or blocks on the read |
| LF-8 | release/entries with an entry signed for another install id, a Keys entry not signed by the anchored customer key, and an entry out of index order | All refused by the agents view, 100 of 100 | Any accepted |
AM3-WP3. Graftyard-run tasks and workspaces
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| LF-9 | Token ceilings by kind: a graftyard-agent token pushes to the base and the fork. A graftyard-workspace token pushes to the base in fork mode. A graftyard_task request from a third App, or for a repository that is neither the base nor its fork | 100 of 100 refused for each | Any accepted |
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| L0-12, Graftyard-run kind | 100 pushes and 100 REST ref writes by a graftyard-workspace token with a live Graftyard-run task, into another holder's Exclusive scope under AgentsExclusive | 100 of 100 of each refused naming the lease, not NoLiveTask | Any accepted, or any refused before the lease check |
| B0-36, Graftyard-run kind | A graftyard-workspace token with a live Graftyard-run task: a REST ref write into another task's Exclusive scope under AgentsExclusive, a push with no live task, and a fork push after its task is cancelled, over git and REST, with shipped and widened permissions | 100 of 100 refused for each, the ref write naming the lease | Any accepted, or the ref write refused before the lease check |
| B0-37, fetch half | A session ref pushed by graftyard-workspace under a live Graftyard-run task, then v0, v1, v2, and --mirror fetches and a v2 ls-refs | The session ref appears only to a v2 ref-prefix that starts with it | It appears to any other listing, or not to the matching ref-prefix |
AM3-WP4. Session logs and session pages
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| LF-10 | Fork-mode session URL in a trailer, read by a base reader with no access to the fork | The page and JSON serve the log from the fork | 404, or the URL names the fork |
AM3-WP5. AM3 gate
| Row | Gate (Pass) |
|---|---|
| LM-3 | A3 end to end in both modes, prompt-only tasks included, the task rules row, containment, ceiling, and agent CI rows rerun, a Sandbox killed mid-task resumes, a bad controller at 1% swaps back, the provenance row passes |
Milestone 6: region loss, SSH, and the lifeboat
M6-WP1. Region loss and re-protection
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-50 | Lose site A with every API token revoked; drain it 24 h; then recover it | Executor-signed ReplaceMember moves every group with a voter on A, using B's and C's registries; RestorePlacement refused until undrained | A group left out, or a restore while drained |
M6-WP2. The SSH front
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-51 | Rotate SSH host keys | UpdateHostKeys clients learn the new set; /api/graftyard/v1/meta/ssh lists both slots for the overlap; same fingerprints on every replica | A mismatch, or a host-key error for an updated client |
| BF-52 | Suspend a user, then push over SSH with their key; separately, revoke an SSH key with 1,000 of its commands queued | Every push from the suspended user refused; 100% of commands decided after the revocation get 401 | Any accepted |
M6-WP3. The lifeboat
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-53 | Lifeboat enroll, a daily LifeboatDecoder, and tail and pack pulls with the R2 feed stopped | Token refused by the main edge; no 7-day alarm; tail and packs fetched from followers | Any leader read, or a main-edge accept |
M6-WP4. Milestone 6 gate
| Row | Gate (Pass) |
|---|---|
| BM-6 | After two release rolls and one FormatsEnable, in a journey 5 drill, one site's member and container classes fail for a day, and ReplaceMember runs with every account API token revoked. It passes only if no script version is created, the spare reads every enabled format, every acknowledged write is present, groups return home, and enrolled gh pr list works against the lifeboat with the edge blocked |
AM4: insights, the predictor, and Code Mode (after M6)
AM4-WP2. The insights consumer and the agents page
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| LF-11 | graftyard apps init on an install with no agents account, then the MCP and A5 insights flows | MCP and overlap check runs work. No Graftyard MCP or graftyard-insights secret exists in any agents account | Either flow needs the agents account |
| LF-14 | A5 replay with every InsightJob delivered twice, plus a stale job after a newer push | One InsightVerdict entry per changed result, none for a repeat, 409 for the stale head, one check run per changed check, and every feed verdict event at an InsightVerdict index | Any duplicate entry or check run, a verdict for a moved head, or a feed verdict event at another index |
| LF-15 | 1,000 agent pushes across a leader change | Exactly one InsightJob per PathSet entry, all from the Insights cursor, none from apply | Any second job for one entry, an entry with no job, or a job from apply |
| LF-16 | PRs 4, 7 and 9, where 4 overlaps 7, 7 overlaps 9, and 4 does not overlap 9. Close 4, then close 7 | overlap:4 on all three, then overlap:7 on 7 and 9, then no overlap: label on 9. Closed PRs keep their labels | Any other name, or a label on an open PR that is in no cluster |
AM4-WP3. Code Mode behind OrgPolicy.code_mode
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| LF-18 | /codemode for an org repository with code_mode off, then on, then off; once for a user-owned repository; a PATCH of code_mode by a non-admin | Refused while off and for the user repository, served while on, refused within 60 s of turning it off, and the non-admin PATCH refused | Any Code Mode call served while off, on a user repository, or 60 s after turning it off, or a non-admin PATCH accepted |
AM4-WP4. AM4 gate
| Row | Gate (Pass) |
|---|---|
| LM-4 | Both predictor rows pass, A5 and A6 pass on a seeded repository, and Code Mode is enabled only if it serves 2026-07-28 |
Milestone 7: breadth and modes
M7-WP3. Managed front and cell accounts
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-66 | 20 isolates hit one key at the limit, then the counter object is killed | x-ratelimit-remaining and enforcement overshoot by at most 20 requests times the number of isolates. The kill refunds at most one window | An overshoot above that bound, or more than one window refunded |
M7-WP4. Child groups for oversized metadata
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-9 | A monorepo member reaching CHILD_GROUP_AT, and octokit.paginate over /repositories/{id} links on a managed cell | The metadata tier moves to a child group with zero SQLITE_FULL; page 2 is served by the right cell | Any SQLITE_FULL, or a page routed to the wrong cell |
| BF-62 | Monorepo member reaches CHILD_GROUP_AT under 300 commands/s with comments, PR opens, merges and releases with tags | Zero SQLITE_FULL. No metadata write gets 503. Every both-tier entry has its MetaHalf within 60 s, with zero lost or doubled. A comment on #n is never delivered before #n opened, 100 of 100. Pre-split feed cursors continue with no 410 | Any SQLITE_FULL, any metadata 503, a lost or doubled half, an out-of-order delivery, or a 410 |
| BF-63 | RepoReset to a t before the split and to a t after it, on a split repository | Both tiers equal a refold at t | Any tier differs |
| LF-17 | Split the metadata tier of a repository that has live InsightVerdict rows | Feed cursors keep working, verdict rows and Verdict events stay on Repo(r), and the check runs and labels reach RepoMeta(r) | Any lost verdict, event, check run or label |
M7-WP5. Solo and workerd for evaluation
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-64 | LeaveEvaluation on a Cloudflare Solo install under write load. Separately, a vendor-signed LeaveEvaluation | System groups and owners end on Tri with moving_from cleared, zero lost acknowledged writes, Placement::solo refused afterwards. The vendor-signed one is refused | Any lost write, a group left on Solo, or the vendor-signed one accepted |
M7-WP6. Import from GHES with the plan preflight
| Row | Measurement | Pass | Fail |
|---|---|---|---|
| BF-65 | GHES archive import with hooks, a workflow, watchers, merged PRs and #N links, on a synthetic ghe-migrator archive built from the documented format while no GHES instance exists inference | Zero webhook deliveries, runs or notifications. Numbers, created_at and authors match. Merged PRs show merged. legacy_id resolves every id | Any delivery, run or notification, or any mismatch |
M7-WP8. Milestone 7 gate
| Row | Gate (Pass) |
|---|---|
| BM-7 | A self-hosted install with no vendor key passes milestones 1 to 6 gates and enables a new format. A vault restore to a chosen cutoff and a lifeboat-export restore into new accounts each lose no entry up to their cutoff, and afterward a developer's PAT and SSH key, a user-owned repository, and an org name all work, while a PAT revoked after the cutoff fails. A Solo group moves to Tri under load with no failed write |