Repository navigation
feat(models): split XS, S, M, and XL writers between two A/B arms - #720
Conversation
Quest-Claim: 7ea8e4437d6ae5b7
quest unblocked --json names each ready quest's arm, picked by a hash of its path, and opens models with that arm's entry; quest models --arm prints an arm's entry for an iterate; Implementer lines record the arm as (arm B); and quest usage rolls up outcomes and spend per size and arm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
There was a problem hiding this comment.
ℹ️ No critical issues — one minor suggestion inline.
Reviewed changes
This review covers the writer A/B split end to end: arm assignment, the quest models --arm surface, Implementer: parsing, the quest usage arm rollup, and the guide and skill edits.
- Arm assignment —
questWritershashes the repository-relative quest path to pick arm A or B. It puts the arm's entry first and drops arms for L and for quests routed up a size by aMaintainer:wait. quest models --arm— prints one arm's entry at the PR's size. It refuses to run together with--writerand refuses L.Implementer:parsing — readsarm Aorarm Bfrom the line's notes, and Fable 5.2 is now a known model.- Usage rollup — the latest-branch tally logic moves into a shared
outcomes().armRowscounts a PR only when it is sized, every line is readable, and every line names the same arm, and it charges every session's spend to that arm. - Guide and skills — iterate dispatch uses
--armfor an armed PR, and/quest-runand/quest-iteraterecord the arm. - Quest tree — deletes
writer-ab.mdand points its links at this PR.
bun run check passes, and the cli, usage, and review suites pass under bun run test.
claude-opus-5-5 | 𝕏
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
This review covers the one commit pushed since the prior Pullfrog review.
- Fallback filter matches on model alone —
questWritersinsrc/cli/models.tsnow drops every entry of the arm's model from the size's list, not only the identical model and effort. M arm B's list is[opus high, sol high], and the earlier thread on this is resolved.
claude-sonnet-5-5 | 𝕏
|
quest land watch: the merge queue removed this pull request at 9abb268 because it conflicts with The land fixer works on it in the background, and the watcher re-enqueues it at the back of the queue once it is fixed or judged flaky, or hands it back here with what it needs. (written by quest land watch) |
|
quest land watch: armed auto-merge on this pull request at 60b01c4, so it joins the back of the merge queue once its required checks pass, with the land fixer's fix ( (written by quest land watch) |
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
This review covers the merge of main (#716, risk tiers) into the branch since the prior Pullfrog review. I checked its conflict resolution with git show --cc.
src/cli/models.ts— keptCLASSIFIERSnext toArm,ARM_NAMES,ARMS, and the armedquestWriterswithout changing either side.src/cli/main.ts— merged the./modelsimport so it includes bothCLASSIFIERSand the arm exports. Thequest usagehelp now lists the writer A/B rollup and then the risk tier rollup.src/cli/usage.ts— keptArmRowandTierRowtogether, soUsagehas botharmsandtiers. In the per-PR text, the risk tier andrisk:line frommainsit alongside the arm notes onimplementer:lines.README.md— combined the two rollup descriptions in thequest usageprose.
tsc --noEmit passes, and the cli, usage, and review suites pass under bun run test (140 tests).
claude-opus-5-5 | 𝕏
Combine #720's writer A/B arm rows with per-cell cost, reviewer rollup and spent.split. A sub-agent known to be in another repository no longer inherits its parent's PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Finishes the writer A/B quest: XS, S, M, and XL quests split 50/50 between two writer arms, each PR records its arm, and
quest usagereports outcomes and spend per arm. The review table insrc/cli/models.tsis untouched.Changes
Arms (
src/cli/models.ts): arm A is the size's own first writer; arm B is the alternative feat(models)!: S writes on Opus medium, A/B the disputed writer picks, and refuse max effort #702 re-armed:A SHA-256 of the quest's path picks the arm, so a restart or respawn keeps it. A quest with an open
Maintainer:decision runs the next size's list and no arm, so an arm always runs at its PR's ownSize:.quest unblocked --jsonaddsarm("A","B", ornull) and opensmodelswith the arm's entry, then the size's list without it. A fallback, such as Sol, or the other arm's model when this arm's is out of quota, still runs, and its PR counts in neither arm.quest models <size> --arm <A|B>prints an arm's entry. An iterate on an armed PR runsquest models <PR size> --arm <arm>, so it stays on the arm's model and effort rather than--writer's effort for the iterate's size. Together with--writerit is a usage error, and on L it fails withL runs no writer arms.Implementer:lines parse anarm Aorarm Bnote, as(arm B)or(iterate, XS, arm B). Fable 5.2 is now a known model.quest usageadds aWriter A/B by size and armrollup (armsin--json). A sized PR counts in it only when everyImplementer:line names the same arm and none is unread. Each row shows PRs, merged, handed back, reviewed, rounds, findings, later## Caused byfixes, and the sessions and spend of every role on those PRs. The per-PR lines show the arm too. The branch-tally logic that accuracy rows already used is now shared with the arm rows.Guide and skills:
/quest-runtells the agent on an armed quest's first entry to record the arm. The guide's iterate dispatch uses--armfor an armed PR and names the arm in the brief, and/quest-iteraterecords it.Quests: deletes
writer-ab.md. Links to it in the routing-evidence epic README,charts.md, andmodel-check.mdnow point to this PR. The epic's readout keeps the XL "report counts" decision and the escapes note, and lists every arm surface the readout must delete.Public API
quest unblocked --jsonobjects gain anarmfield, and armed sizes'modelslists gain the arm's entry first (XS B and S B now list three entries).quest modelsgains--arm <A|B>.quest usage --jsongainsarms, and each PR'simplementersentries gainarm.Validation
just checkandjust testpass: 1237 tests, plus the install, release, and lifecycle scripts.dist/quest models M --arm Bprints Opus high,models XL --arm B --jsonprints Fable medium, andmodels L --arm Aexits 2 withL runs no writer arms.sha256(path)[0] & 1and compared it withdist/quest unblocked --jsonon this repository's tree. All 11 matched (3 A, 7 B, 1 unarmed L). After the review fix, the built CLI givesquest/m0/benchmark/risk-tier.md, which has aMaintainer:wait,arm: null. In tests, 41 fixture paths split 21/20.dist/quest usage --since 2026-11-12 --pr 702reads a real PR without error. token-max is not installed on this host, so no spend was shown. The fixture test covers the arm rollup's spend, rounds, and exclusions: a fallback, mixed arms, and an unsized PR.Choices made without asking
modelslist.assets/AGENTS.md,assets/skills/run.md, andassets/skills/iterate.md. The quest's Goal needs dispatchers and iterates to record arms, and that is only possible there.Maintainer:decision runs no arm. Its cell would otherwise be ambiguous: iterates and the rollup read the PR's own size, as the review's first round found. The cost is that a few quests drop out of the sample.arm A. The quest's path hashes to A, and Opus medium is M's arm-A entry.Decisions
assets/AGENTS.md,/quest-runand/quest-iterateedits (rejected: reject them)src/cli/models.ts:87: an arm's entry now drops every other entry of the same model, not only the identical one, since a same-model fallback hits the same quota. M B's list is[opus high, sol high].Follow-ups
None.
Models
🤖 Generated with Claude Code
(Written by Claude Opus 5.5 with medium)
Review
mechanical fix by codex, gpt-6.1-sol, medium effort:
Merged base and mechanically re·solved README.md, src/cli/main.ts, src/cli/models.ts, and src/cli/usage.ts conflicts, preserving writer A/B and risk reporting; ran nix develop --command just install, nix develop --command bunx prettier --write README.md src/cli/main.ts src/cli/models.ts src/cli/usage.ts, nix develop --command just check, and nix develop --command just test.(head 60b01c4)Rewritten by
quest reviewafter each round.Head: c39181b
Base: origin/main at d0f289d
Reviewer: Codex, gpt-6.1-sol, high effort
Session:
01a1d768-5891-7462-a0d4-d78f07b3ec1f, round 2Score: 10/10,
The routing fix resolves the earlier finding, and the incremental changes introduce no new findings.Rounds: 2; findings 1 (medium 1): fixed 1, refuted 0, open 0
Findings: none
The last round's findings, at 607278e:
src/cli/models.ts:85:Preserve the effective writer size for armed iterates;src/cli/models.ts:81-82 now returns arm: null and the next size's writer list for Maintainer waits, so every assigned arm uses the PR's own size.Reviewer: Codex, gpt-6.1-sol, medium effort, shadow
Session:
01a1d768-5891-76e3-b5c8-0d43566c30daRounds: 1; findings 0: fixed 0, refuted 0, open 0
9abb268 applies Pullfrog's suggested patch on
src/cli/models.ts:87unchanged on top of the reviewed head c39181b, which/quest-mergeexempts from a new round.