# Three leads, one plan — can a codex model hold the plan-lead chair?

slug: `plan-lead-abc` · status: **in progress — calibration arm pending** · started 2026-10-25 · players: terra, astra (vs opus)

## Question

SPINDLE's plan phase is led by Opus today; codex models (terra, astra) already lead build and review. Can gpt-5.6-terra
(xhigh) or gpt-6-astra (high) take the **plan-lead** chair — orchestrating discovery, the owner interview, the two blind
co-drafters, the seed review and the convergence loop — and produce a plan as good as the Opus-led plan for the same unit?
"As good" is not "the same": the plan is scored against what the real build later proved mattered, and over-rigor past
the unit-sizing ceiling counts against it.

## Summary (provisional)

Seven codex-led plan runs on three already-shipped units, in a sealed environment, scored blind against frozen rubrics
mined from each unit's real build history. Rubric means: **Opus historical 93%, astra 58%, terra 43%** (n = 3 / 3 / 4).
Both codex arms produced coherent, committed, manifest-backed plans on every run, in 76–194 minutes, with every seat live
(no replay). But they systematically mis-sized the work against what was actually built — under-scoping DEFUSE-CF and
CANCEL-SETTLEMENT (whole tiers or production surfaces dropped) and over-mechanizing RESEARCH-RECLAIM (lock protocols and
ledgers the shipped unit never needed). Astra beat terra on every unit but needed a human at the start of every run
(3/3 prerequisite stops) and between steps; terra ran through with one stop in five runs and the fastest cycle (82 min),
which was also the lowest-scoring plan (32.8%). The Opus number is inflated by construction (the rubric was mined from the
build that followed that plan), so the honest headline waits for the Opus-fresh calibration arm, held for a quota reset.

## Method

### Units
| unit | repo | label | code diff shipped | rounds shipped |
|---|---|---|---|---|
| DEFUSE-CF | scriptgen | S | +556 / 14 files | plan 3 · P4 3 · checkpoints 11 |
| RESEARCH-RECLAIM | scriptgen | M | +4,455 / 32 files | plan 3 · P4 3 · checkpoints 18 |
| LLM-PROXY-CANCEL-SETTLEMENT | flowmaster | S–M (money/consult) | +1,057 / 8 files | plan 3 · P4 3 · checkpoints 13 |

### Arms
- **A — Opus 5.5 @ xhigh (historical):** the plan as committed at its plan-lock commit (DEFUSE 724bfa3b, RR 90f39227,
  CANCEL e6cebab9a), produced before the build by the production pipeline. **A-fresh** (Opus re-run in the bench) is
  pending.
- **B — gpt-5.6-terra @ xhigh** · **C — gpt-6-astra @ high:** same pipeline, only the lead seat changed. Codex plan
  templates were translated from the Opus ones (`plan-codex.md`, `planloop-codex.md`): same discovery fleet (GLM lattice
  4–6 lanes + merge, two Opus deep seats, Opus + codex web seats), same owner interview through the `needs_human` flag,
  Fable-MAX + astra blind co-drafters, lead red-team + synthesis, sol seed, terra plan-loop rounds, STEP_MANIFEST, plan
  lock. Every seat is a live bridge call with a run record (model from the run's metadata, brief hash, answer hash) in a
  per-run SEAT_LEDGER; replaying or reading a prior run's seat output is forbidden by the template.

### Isolation
- A separate WSL distro with Windows drives and interop off, no `gh`, GitHub hostnames blackholed, non-sudo user, no
  memory index, repo-local CLAUDE.md absent (see limitations), account rules placed above the repos.
- Each arm gets its own main-only clone of each unit repo rewound to the commit **before** the unit's docs directory
  appeared, other refs dropped, objects pruned; verified `git cat-file -e <merge sha>` fails in every clone.
- Forensics over all 40 codex rollouts (1,979 executed commands): 0 reads of the real merge SHAs; 0 network commands to
  non-provider hosts; 0 credential/token reads; 0 reads of another run's repo or worktree. Residuals: astra probed the
  hultra-prescribed `/mnt/c` path twice (nonexistent); astra RR ran one `find` for a project-memory index (none present);
  one web seat searched GitHub for the unit by name (private repo, nothing returned); astra used codex's built-in
  `spawn_agent` once to write a blocker artifact (an unaccounted sub-agent, outside the seat roster).

### Scoring
1. **Frozen rubric (primary, objective).** Before any bench plan was read, one Opus worker per unit mined 24 / 29 / 29
   items of "what actually mattered" from the real build: must/should review findings and their fixes, DECISIONS rulings
   that changed the approach, escalations, checkpoint failures, post-merge fixes. Each item carries a scoring note
   (ANTICIPATES / PARTIAL / MISSES). Weighted % = anticipates + ½·partial.
2. **Blind scoring.** Plan + STEP_MANIFEST + DECISIONS per arm, with every model and vendor name redacted to tokens,
   shuffled to P1–P10; one Opus scorer per plan, instructed not to infer authorship.
3. **Sizing axis.** Steps, declared `## SIZING` projection, unsettled forks, and scope vs the shipped truth (under / match /
   over). Over-rigor past the guide ceiling is a defect.
4. **Interviews** were answered by the PM as the owner's proxy from a frozen key of the owner's historical answers
   (recommended option; consults as vetted; only Recommended enhancements; the historical ruling when a topic matched).
5. Blind cross-family panel: **not yet run** (deferred for Claude quota).

## Results

### Rubric scores (weighted %, A/P/M of N)
| unit | A opus-hist | B terra | C astra |
|---|---|---|---|
| DEFUSE-CF | **97.9** (23/1/0) | 37.5 (4/10/10) · pilot | 54.2 (9/8/7) · pilot |
| CANCEL-SETTLEMENT | **94.8** (26/3/0) | 50.0 (9/11/9) | 63.8 (12/13/4) |
| RESEARCH-RECLAIM | **86.2** (21/8/0) | 51.7 (7/16/6) · pilot · 32.8 (0/19/10) · measured | 56.9 (8/17/4) |

Dropping the test/verification items moves every score by ≤2 points (the test-freeze handicap, below, does not explain
the gap).

### Sizing vs truth
| unit | A | B | C |
|---|---|---|---|
| DEFUSE-CF | 7 steps · match | 6 · **under** (one tier, no strict list, no golden, no gate) | 5 · **under** (strict tier declined) |
| CANCEL-SETTLEMENT | 8 · match | 7 · **under** (no pg_cron gate, no recovery driver, no mark timeout) | 8 · **over** in rigor (7/8 heavy fences, second test harness) · under on 4 surfaces |
| RESEARCH-RECLAIM | 10 · match | 7 · **over**-mechanized (file ledger, v2 manifest reconstruction; dropped the DB record) · 9 · **over** (handle-relative delete stack) | 8 · **over** (cross-process lock protocol) |

Every codex arm wrote a `## SIZING` section with a projected code mass; none of the historical plans had one (the guide
pointer was spliced into the templates only this week). Astra's projections bracketed the truth; terra's were plausible
but the plans still mis-scoped: the declared forks were "0" in every codex plan while scorers counted 2–4 latent ones.

### Process
| run | unit · arm | lead minutes | live seats | lead stops | PM infra fixes |
|---|---|---|---|---|---|
| 1 | DEFUSE-CF · terra (pilot) | 76 | 21 | 0 | 2 |
| 2 | DEFUSE-CF · astra (pilot) | 81 | 23 | 1 | 0 |
| 3 | RESEARCH-RECLAIM · terra (pilot) | 99 | 21 | 0 | 0 |
| 4 | CANCEL-SETTLEMENT · terra | 148 | 22 | 0 | 2 |
| 5 | RESEARCH-RECLAIM · astra | 194 | 25 | 1 (+2 idle nudges) | 2 |
| 6 | CANCEL-SETTLEMENT · astra | 136 | 22 | 1 | 1 |
| 7 | RESEARCH-RECLAIM · terra (measured) | 82 | 15 | 1 | 1 |

- **Astra stops at prerequisites; terra does not.** All three astra cells opened with a hard stop on hultra's
  `ultra_prompt.md` rule and absent repo instruction files, and astra idled between steps (post-seed, post-loop) until
  nudged. Terra noted the same gaps and proceeded; its one stop was impatience (escalating on a running co-drafter).
- **Lead stops are not quality.** Astra's extra stops cost ~40–110 minutes per run and did not buy a better plan than its
  own discipline did; terra's fastest run (82 min, 15 seats) was its worst plan.
- **Seat usage was full and real** (21–25 live seats per run; the 15-seat run skipped nothing mandatory). The ledgers
  record Fable, Opus, Sonnet, GLM, sol, terra and astra calls by actual model id. Both codex leads briefed their seats
  themselves; how well they did so is inside the scores, not separated here.
- **Planning speed:** codex plan runs took 76–194 minutes end to end; the historical Opus plan phases ran for hours
  (uninstrumented, not comparable — see limitations).

### What the misses were (recurring)
- Treating the account-level test freeze as binding in a repo that has none → plans with no build gate and no tests
  (terra DEFUSE-CF, terra RR-measured). Real defect in the plans; also an environment handicap (below).
- Dropping a whole tier or surface the real build needed: DEFUSE-CF's strict/DICP list and evidence-fragment ingress
  (both codex arms); CANCEL's pg_cron scheduling and server-side bounds (both arms); RR's terminal `finished` trigger and
  migration pins (terra and astra).
- Over-building mechanism where the shipped unit used an existing seam: RR cross-process locks, handle-relative delete
  stacks, file ledgers.
- Internal contradictions scorers caught: wall-capped cleanup vs guaranteed release (CANCEL terra); retained
  `transform.json` yet "unknown extractor" degradation (RR terra-measured).

## Limitations — read before the numbers
1. **The historical Opus score is inflated by construction.** The rubric was mined from the build that *followed* the
   historical plan, so the plan that defined that build naturally "anticipates" it. The Opus-fresh arm (same bench,
   same inputs, same proxy) is the control; it is pending for Claude-quota reasons. Until it runs, the 93 is an upper
   bound, not a comparison.
2. **Missing repo-local instructions.** The bench repos did not carry scriptgen's gitignored `CLAUDE.md` ("no test
   freeze; T3 runs tests"); codex leads inherited the account-level freeze. Sensitivity: ≤2 points on the rubric, but the
   historical lead had that file.
3. **Pilot-cycle faults (runs 1–3):** run 1 received no consult contract (the splice needs an `origin/main`, which the
   isolation step had removed); GLM lanes ran read-blocked (no `bwrap`) in runs 1–3; a copied Claude login expired for
   25 minutes during runs 4–5 (three astra seats failed and were re-run). Fixed before runs 6–7.
4. **Proxy consistency:** the owner's "only `original.bin`" granularity ruling for RESEARCH-RECLAIM was injected into the
   terra re-run's interview (whose questions assumed whole-bundle deletion) but not into astra's; astra's RR-01 miss is
   partly the proxy's.
5. **Single scorer per plan**, same model family as arm A; no inter-rater check yet. Rubric items are not weighted.
6. **n is small** (3 units, 7 runs); run-to-run variance is visible (terra RR 51.7 pilot vs 32.8 measured with the same
   inputs). Treat per-unit gaps as the signal, single numbers as illustration.

## Verdict (provisional)
- **Headline:** neither codex model earned the plan-lead chair in this cycle. Astra plans are consistently the better
  codex plans (58 vs 43) and read as careful, but both arms mis-size the unit against reality more often than not, and
  the historical Opus plans, with the caveat above, matched the shipped scope every time.
- **Not yet:** a lead that under-scopes a money path (CANCEL: no recovery driver) or drops the strict tier of a
  sanitizer (DEFUSE-CF) produces plans that would have cost the review rounds the sizing guide warns about.
- **Path:** (1) run the Opus-fresh control; (2) re-run one unit with the repo-local instructions present; (3) add the
  blind panel and a second scorer; (4) if the gap holds, test astra as plan lead *with* an Opus synthesis check-in
  before lock (the codex-lead rail already exists for build), rather than as a solo chair.
- **Harness lessons regardless of verdict:** the `UNIT_SIZING_GUIDE` pointer was missing from every plan template
  (fixed fleet-wide during this study); codex leads need the bench's two template/daemon fixes (`/context` probe,
  stop-after-plan) if they ever run plan/planloop; the consult splice must tolerate repos without a remote.

## Sources
- Run logs with every intervention (UTC): `bench/runs/run<N>_<UNIT>_<arm>.log`
- Per-run harvests: plan, manifest, DECISIONS, co-plan drafts with provenance, discovery, review rounds, SEAT_LEDGER.ndjson, daemon logs
- Rubrics: `bench/rubric_<UNIT>.md`; interview key: `bench/interview_keys.md`; template translation notes: `bench/TEMPLATE_NOTES.md`
- Scores: `bench/scores/P1..P10.{md,json}`; blind key: `bench/blind/KEY.tsv`; aggregate: `bench/SUMMARY.md`, `summary.json`
- Forensics: `bench/forensics.txt` (text level), `bench/forensics_commands.txt` (command level)