Study 09 · started 2026-10-25Provisional · calibration arm pending

Can a codex model hold the plan-lead chair?

SPINDLE's plan phase is led by Opus today; codex models (terra, astra) already lead build and review. Can gpt-5.6-terra (xhigh) or gpt-6-astra (high) take the plan-lead chair — orchestrating discovery, the owner interview, the two blind co-drafters, the seed review and the convergence loop — and produce a plan as good as the Opus-led plan for the same unit? "As good" is not "the same": the plan is scored against what the real build later proved mattered, and over-rigor past the unit-sizing ceiling counts against it.

Read before the numbers — six limitations, the first one load-bearing

This is a provisional study. The Opus arm below is the historical plan, scored against a rubric mined from the build that followed it — so its score is an upper bound by construction. The Opus-fresh control (same bench, same inputs, same proxy) is pending for Claude quota. Nothing on this page should be read as a comparison until it runs.

01The historical Opus score is inflated by construction. The rubric was mined from the build that followed the historical plan, so the plan that defined that build naturally "anticipates" it. The Opus-fresh arm (same bench, same inputs, same proxy) is the control; it is pending for Claude-quota reasons. Until it runs, the 93 is an upper bound, not a comparison.
02Missing repo-local instructions. The bench repos did not carry scriptgen's gitignored CLAUDE.md ("no test freeze; T3 runs tests"); codex leads inherited the account-level freeze. Sensitivity: ≤2 points on the rubric, but the historical lead had that file.
03Pilot-cycle faults (runs 1–3). Run 1 received no consult contract (the splice needs an origin/main, which the isolation step had removed); GLM lanes ran read-blocked (no bwrap) in runs 1–3; a copied Claude login expired for 25 minutes during runs 4–5 (three astra seats failed and were re-run). Fixed before runs 6–7.
04Proxy consistency. The owner's "only original.bin" granularity ruling for RESEARCH-RECLAIM was injected into the terra re-run's interview (whose questions assumed whole-bundle deletion) but not into astra's; astra's RR-01 miss is partly the proxy's.
05Single scorer per plan, same model family as arm A. No inter-rater check yet. Rubric items are not weighted.
06n is small (3 units, 7 runs). Run-to-run variance is visible (terra RR 51.7 pilot vs 32.8 measured with the same inputs). Treat per-unit gaps as the signal, single numbers as illustration.

Neither codex model earned the plan-lead chair in this cycle. Astra plans are consistently the better codex plans (58 vs 43) and read as careful, but both arms mis-size the unit against reality more often than not, and the historical Opus plans, with the caveat above, matched the shipped scope every time.

Seven codex-led plan runs on three already-shipped units, in a sealed environment, scored blind against frozen rubrics mined from each unit's real build history. Rubric means: Opus historical 93%, astra 58%, terra 43% (n = 3 / 3 / 4). Both codex arms produced coherent, committed, manifest-backed plans on every run, in 76–194 minutes, with every seat live (no replay). But they systematically mis-sized the work against what was actually built — under-scoping DEFUSE-CF and CANCEL-SETTLEMENT (whole tiers or production surfaces dropped) and over-mechanizing RESEARCH-RECLAIM (lock protocols and ledgers the shipped unit never needed). Astra beat terra on every unit but needed a human at the start of every run (3/3 prerequisite stops) and between steps; terra ran through with one stop in five runs and the fastest cycle (82 min), which was also the lowest-scoring plan (32.8%). The Opus number is inflated by construction (the rubric was mined from the build that followed that plan), so the honest headline waits for the Opus-fresh calibration arm, held for a quota reset.

Opus · historicaln = 3

93%

rubric mean · inflated by construction — upper bound until A-fresh runs

astran = 3

58%

rubric mean · weighted % = anticipates + ½·partial

terran = 4

43%

rubric mean · weighted % = anticipates + ½·partial

The method

sealed bench, frozen rubrics, blind scores

Arm A · Claude Opus 5.5 @ xhigh

The plan as committed at its plan-lock commit (DEFUSE 724bfa3b, RR 90f39227, CANCEL e6cebab9a), produced before the build by the production pipeline.

Arm B · gpt-5.6-terra @ xhigh

Same pipeline, only the lead seat changed. Codex plan templates were translated from the Opus ones (plan-codex.md, planloop-codex.md).

Arm C · gpt-6-astra @ high

Same pipeline, only the lead seat changed. Same translated templates as arm B.

Arm A-fresh · pending

A-fresh — Opus re-run in the bench, same inputs, same proxy — is pending for Claude-quota reasons. It is the control; until it runs, the historical Opus score is an upper bound, not a comparison.

Same discovery fleet (GLM lattice 4–6 lanes + merge, two Opus deep seats, Opus + codex web seats), same owner interview through the needs_human flag, Fable-MAX + astra blind co-drafters, lead red-team + synthesis, sol seed, terra plan-loop rounds, STEP_MANIFEST, plan lock. Every seat is a live bridge call with a run record (model from the run's metadata, brief hash, answer hash) in a per-run SEAT_LEDGER; replaying or reading a prior run's seat output is forbidden by the template.

unit	repo	label	code diff shipped	rounds shipped	rubric items
DEFUSE-CF	scriptgen	S	+556 / 14 files	plan 3 · P4 3 · checkpoints 11	24
RESEARCH-RECLAIM	scriptgen	M	+4,455 / 32 files	plan 3 · P4 3 · checkpoints 18	29
LLM-PROXY-CANCEL-SETTLEMENT	flowmaster	S–M (money/consult)	+1,057 / 8 files	plan 3 · P4 3 · checkpoints 13	29

How a plan is scored

declared before any bench plan was read

Frozen rubric (primary, objective)

Before any bench plan was read, one Opus worker per unit mined 24 / 29 / 29 items of "what actually mattered" from the real build: must/should review findings and their fixes, DECISIONS rulings that changed the approach, escalations, checkpoint failures, post-merge fixes. Each item carries a scoring note (ANTICIPATES / PARTIAL / MISSES). Weighted % = anticipates + ½·partial.

Blind scoring

Plan + STEP_MANIFEST + DECISIONS per arm, with every model and vendor name redacted to tokens, shuffled to P1–P10; one Opus scorer per plan, instructed not to infer authorship.

Sizing axis

Steps, declared ## SIZING projection, unsettled forks, and scope vs the shipped truth (under / match / over). Over-rigor past the guide ceiling is a defect.

Interviews by proxy

Answered by the PM as the owner's proxy from a frozen key of the owner's historical answers (recommended option; consults as vetted; only Recommended enhancements; the historical ruling when a topic matched).

Blind cross-family panel

Not yet run (deferred for Claude quota).

Rubric scores, per unit and arm

weighted % · (anticipates / partial / misses)

Each cell is one blind-scored plan (P1–P10). Arm A is the historical Opus plan and carries the construction inflation described above. RESEARCH-RECLAIM has two terra cells: the pilot run and the measured re-run with the same inputs. Dropping the test/verification items moves every score by ≤2 points — the test-freeze handicap (limitation 2) does not explain the gap.

unit	A · Opus · historical	B · terra	C · astra
DEFUSE-CFof 24 items	97.9(23/1/0) historicalP5	37.5(4/10/10) pilotP1	54.2(9/8/7) pilotP4
RESEARCH-RECLAIMof 29 items	86.2(21/8/0) historicalP2	51.7(7/16/6) pilotP7 32.8(0/19/10) measuredP10	56.9(8/17/4) measuredP8
LLM-PROXY-CANCEL-SETTLEMENTof 29 items	94.8(26/3/0) historicalP9	50.0(9/11/9) measuredP6	63.8(12/13/4) measuredP3

weighted % = anticipates + ½·partial over the unit's rubric items · pilot = runs 1–3 (pilot-cycle faults, limitation 3) · measured = runs 4–7

Sizing vs truth

steps · scope against the shipped unit

Over-rigor past the sizing-guide ceiling is a defect, the same as under-scoping. Match means the plan's steps covered what the real build shipped, no more.

unit	A · Opus · historical	B · terra	C · astra
DEFUSE-CF	match 7 steps · match	under 6 · under (one tier, no strict list, no golden, no gate)	under 5 · under (strict tier declined)
LLM-PROXY-CANCEL-SETTLEMENT	match 8 · match	under 7 · under (no pg_cron gate, no recovery driver, no mark timeout)	over 8 · over in rigor (7/8 heavy fences, second test harness) · under on 4 surfaces
RESEARCH-RECLAIM	match 10 · match	overover 7 · over-mechanized (file ledger, v2 manifest reconstruction; dropped the DB record) · 9 · over (handle-relative delete stack)	over 8 · over (cross-process lock protocol)

·Every codex arm wrote a ## SIZING section with a projected code mass; none of the historical plans had one (the guide pointer was spliced into the templates only this week).
·Astra's projections bracketed the truth; terra's were plausible but the plans still mis-scoped: the declared forks were "0" in every codex plan while scorers counted 2–4 latent ones.

How the runs went

from the run logs

Process metrics are published for inspection, not scored. Lead minutes are end to end; a stop is the lead handing control to a human.

run	unit · arm	lead minutes	live seats	lead stops	PM infra fixes	note
1	DEFUSE-CF · terrapilot	76	21	0	2	no consult (origin missing at dispatch); GLM lanes read-blocked (no bwrap)
2	DEFUSE-CF · astrapilot	81	23	1	0	prereq stop at 3 min; GLM lanes read-blocked (no bwrap)
3	RESEARCH-RECLAIM · terrapilot	99	21	0	0	GLM lanes read-blocked (no bwrap)
4	LLM-PROXY-CANCEL-SETTLEMENT · terrameasured	148	22	0	2	~25 min Claude auth outage inside the run; /context composer stall
5	RESEARCH-RECLAIM · astrameasured	194	25	1 (+2 idle nudges)	2	identity+auth stop (infra); 2 idle nudges; proxy omitted granularity ruling
6	LLM-PROXY-CANCEL-SETTLEMENT · astrameasured	136	22	1	1	prereq stop at 5 min; /context composer stall
7	RESEARCH-RECLAIM · terrameasured	82	15	1	1	escalated on a running co-drafter (impatience); /context composer stall

Astra stops at prerequisites; terra does not. All three astra cells opened with a hard stop on hultra's ultra_prompt.md rule and absent repo instruction files, and astra idled between steps (post-seed, post-loop) until nudged. Terra noted the same gaps and proceeded; its one stop was impatience (escalating on a running co-drafter).

Lead stops are not quality. Astra's extra stops cost ~40–110 minutes per run and did not buy a better plan than its own discipline did; terra's fastest run (82 min, 15 seats) was its worst plan.

Seat usage was full and real (21–25 live seats per run; the 15-seat run skipped nothing mandatory). The ledgers record Fable, Opus, Sonnet, GLM, sol, terra and astra calls by actual model id. Both codex leads briefed their seats themselves; how well they did so is inside the scores, not separated here.

Planning speed: codex plan runs took 76–194 minutes end to end; the historical Opus plan phases ran for hours (uninstrumented, not comparable — see limitations).

What the misses were

recurring, both arms

01Treating the account-level test freeze as binding in a repo that has none → plans with no build gate and no tests (terra DEFUSE-CF, terra RR-measured). Real defect in the plans; also an environment handicap (limitation 2).
02Dropping a whole tier or surface the real build needed: DEFUSE-CF's strict/DICP list and evidence-fragment ingress (both codex arms); CANCEL's pg_cron scheduling and server-side bounds (both arms); RR's terminal finished trigger and migration pins (terra and astra).
03Over-building mechanism where the shipped unit used an existing seam: RR cross-process locks, handle-relative delete stacks, file ledgers.
04Internal contradictions scorers caught: wall-capped cleanup vs guaranteed release (CANCEL terra); retained transform.json yet "unknown extractor" degradation (RR terra-measured).

Did the isolation hold?

forensics, disclosed

Held

·A separate WSL distro with Windows drives and interop off, no gh, GitHub hostnames blackholed, non-sudo user, no memory index, repo-local CLAUDE.md absent (see limitations), account rules placed above the repos.
·Each arm gets its own main-only clone of each unit repo rewound to the commit before the unit's docs directory appeared, other refs dropped, objects pruned; verified git cat-file -e <merge sha> fails in every clone.
·Forensics over all 40 codex rollouts (1,979 executed commands): 0 reads of the real merge SHAs; 0 network commands to non-provider hosts; 0 credential/token reads; 0 reads of another run's repo or worktree.

Residuals

·Astra probed the hultra-prescribed /mnt/c path twice (nonexistent).
·Astra RR ran one find for a project-memory index (none present).
·One web seat searched GitHub for the unit by name (private repo, nothing returned).
·Astra used codex's built-in spawn_agent once to write a blocker artifact — an unaccounted sub-agent, outside the seat roster.

Verdict (provisional)

Provisional · calibration arm pending

Not yet. A lead that under-scopes a money path (CANCEL: no recovery driver) or drops the strict tier of a sanitizer (DEFUSE-CF) produces plans that would have cost the review rounds the sizing guide warns about.

The path. (1) Run the Opus-fresh control; (2) re-run one unit with the repo-local instructions present; (3) add the blind panel and a second scorer; (4) if the gap holds, test astra as plan lead with an Opus synthesis check-in before lock (the codex-lead rail already exists for build), rather than as a solo chair.

Opus · historical · n 3: 93%
astra · n 3: 58%
terra · n 4: 43%

rubric mean · Opus historical is an upper bound until A-fresh runs

Harness lessons regardless of verdict: The UNIT_SIZING_GUIDE pointer was missing from every plan template (fixed fleet-wide during this study); codex leads need the bench's two template/daemon fixes (/context probe, stop-after-plan) if they ever run plan/planloop; the consult splice must tolerate repos without a remote.

Pending before this verdict is final: Opus-fresh calibration arm (A-fresh): same bench, same inputs, same proxy — pending for Claude quota; until it runs the historical 93 is an upper bound · blind cross-family panel: not yet run (deferred for Claude quota) · second scorer / inter-rater check: single Opus scorer per plan so far · repo-local-instructions re-run: one unit re-run with scriptgen's gitignored CLAUDE.md present.

What changed in the harness because of it

01The UNIT_SIZING_GUIDE pointer was missing from every plan template; it was spliced into the plan templates fleet-wide during this study. Every codex arm wrote a `## SIZING` section as a result; none of the historical plans had one.
02Codex plan and plan-loop templates (`plan-codex.md`, `planloop-codex.md`) were translated from the Opus ones with two template/daemon fixes (`/context` probe, stop-after-plan) that codex leads need if they ever run plan or planloop in production.
03Found, not yet fixed: the consult splice needs an `origin/main` and must learn to tolerate repos without a remote — run 1 received no consult contract because the isolation step had removed it.

Raw data

generated 2026-10-26 by scripts/schemas/plan-lead-abc.ts

Every number on this page is read from data/plan-lead-abc.json, which the generator rewrites from these source files. A data update is a commit.

/raw/plan-lead-abc/STUDY.mdThe provisional study as written at the bench: question, method, every table, the six limitations, the verdict. This page is a transcription of it.
/raw/plan-lead-abc/summary.jsonMachine-readable scores per blind plan P1–P10: weighted %, A/P/M of N, steps, SIZING section, scope, run, cycle, lead minutes, seats, stops, infra fixes.
bench/runs/run<N>_<UNIT>_<arm>.logRun logs with every intervention (UTC). Held at the bench.
bench/harvest/Per-run harvests: plan, manifest, DECISIONS, co-plan drafts with provenance, discovery, review rounds, SEAT_LEDGER.ndjson, daemon logs. Held at the bench.
bench/rubric_<UNIT>.mdThe frozen rubrics (24 / 29 / 29 items), mined from each unit's real build before any bench plan was read; interview key in bench/interview_keys.md; template translation notes in bench/TEMPLATE_NOTES.md.
bench/scores/P1..P10.{md,json}Per-plan blind scores with per-item evidence; blind key bench/blind/KEY.tsv; aggregate bench/SUMMARY.md.
bench/forensics.txtIsolation forensics over all 40 codex rollouts — text level; bench/forensics_commands.txt at command level (1,979 executed commands).

Deliberately unfilled: Opus-fresh calibration arm (A-fresh): same bench, same inputs, same proxy — pending for Claude quota; until it runs the historical 93 is an upper bound · blind cross-family panel: not yet run (deferred for Claude quota) · second scorer / inter-rater check: single Opus scorer per plan so far · repo-local-instructions re-run: one unit re-run with scriptgen's gitignored CLAUDE.md present.