Skip to content

feat(models): split XS, S, M, and XL writers between two A/B arms - #720

Merged
moq-bot[bot] merged 5 commits into
mainfrom
quest/m0/routing-evidence/writer-ab
Nov 14, 2026
Merged

moq-bot[bot] merged 5 commits into
mainfrom
quest/m0/routing-evidence/writer-ab

Conversation

@kixelated

@kixelated kixelated commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes the writer A/B quest: XS, S, M, and XL quests split 50/50 between two writer arms, each PR records its arm, and quest usage reports outcomes and spend per arm. The review table in src/cli/models.ts is untouched.

Changes

  • Arms (src/cli/models.ts): arm A is the size's own first writer; arm B is the alternative feat(models)!: S writes on Opus medium, A/B the disputed writer picks, and refuse max effort #702 re-armed:

    Size A B
    XS Sonnet 5.5 high Opus 5.5 medium
    S Opus medium Sonnet high
    M Opus medium Opus high
    L no arm no arm
    XL Opus xhigh Fable 5.2 medium

    A SHA-256 of the quest's path picks the arm, so a restart or respawn keeps it. A quest with an open Maintainer: decision runs the next size's list and no arm, so an arm always runs at its PR's own Size:.

  • quest unblocked --json adds arm ("A", "B", or null) and opens models with the arm's entry, then the size's list without it. A fallback, such as Sol, or the other arm's model when this arm's is out of quota, still runs, and its PR counts in neither arm.

  • quest models <size> --arm <A|B> prints an arm's entry. An iterate on an armed PR runs quest models <PR size> --arm <arm>, so it stays on the arm's model and effort rather than --writer's effort for the iterate's size. Together with --writer it is a usage error, and on L it fails with L runs no writer arms.

  • Implementer: lines parse an arm A or arm B note, as (arm B) or (iterate, XS, arm B). Fable 5.2 is now a known model.

  • quest usage adds a Writer A/B by size and arm rollup (arms in --json). A sized PR counts in it only when every Implementer: line names the same arm and none is unread. Each row shows PRs, merged, handed back, reviewed, rounds, findings, later ## Caused by fixes, and the sessions and spend of every role on those PRs. The per-PR lines show the arm too. The branch-tally logic that accuracy rows already used is now shared with the arm rows.

  • Guide and skills: /quest-run tells the agent on an armed quest's first entry to record the arm. The guide's iterate dispatch uses --arm for an armed PR and names the arm in the brief, and /quest-iterate records it.

  • Quests: deletes writer-ab.md. Links to it in the routing-evidence epic README, charts.md, and model-check.md now point to this PR. The epic's readout keeps the XL "report counts" decision and the escapes note, and lists every arm surface the readout must delete.

Public API

  • quest unblocked --json objects gain an arm field, and armed sizes' models lists gain the arm's entry first (XS B and S B now list three entries).
  • quest models gains --arm <A|B>.
  • quest usage --json gains arms, and each PR's implementers entries gain arm.
  • None of this breaks existing callers: every change adds a field or option.

Validation

  • just check and just test pass: 1237 tests, plus the install, release, and lifecycle scripts.
  • Built CLI: dist/quest models M --arm B prints Opus high, models XL --arm B --json prints Fable medium, and models L --arm A exits 2 with L runs no writer arms.
  • Independent check: I recomputed each ready quest's arm in Node from sha256(path)[0] & 1 and compared it with dist/quest unblocked --json on this repository's tree. All 11 matched (3 A, 7 B, 1 unarmed L). After the review fix, the built CLI gives quest/m0/benchmark/risk-tier.md, which has a Maintainer: wait, arm: null. In tests, 41 fixture paths split 21/20.
  • dist/quest usage --since 2026-11-12 --pr 702 reads a real PR without error. token-max is not installed on this host, so no spend was shown. The fixture test covers the arm rollup's spend, rounds, and exclusions: a fallback, mixed arms, and an unsized PR.

Choices made without asking

  • A is the size's own table entry and B is the alternative, so the readout's "keep A" means changing nothing.
  • An iterate runs the arm's entry at the PR's size, not an effort set by the iterate's size. Without this, an M arm-B PR's small iterate would drop to Opus medium and blur the effort comparison. The iterate's line still records the size it was given.
  • A writer that falls back to the other arm's model names no arm, so its PR counts in neither. That model is not first in its models list.
  • The arm rollup charges every session's spend on the PR (writing, iterating, and reviewing) to the arm, since rounds differ between arms.
  • I edited assets/AGENTS.md, assets/skills/run.md, and assets/skills/iterate.md. The quest's Goal needs dispatchers and iterates to record arms, and that is only possible there.
  • A quest routed up a size for an open Maintainer: decision runs no arm. Its cell would otherwise be ambiguous: iterates and the rollup read the PR's own size, as the review's first round found. The cost is that a few quests drop out of the sample.
  • This PR's own Implementer line records arm A. The quest's path hashes to A, and Opus medium is M's arm-A entry.

Decisions

  • ✅ iterates on an armed PR run the arm's effort for the PR's size (rejected: size by work left)
  • ✅ accept the assets/AGENTS.md, /quest-run and /quest-iterate edits (rejected: reject them)
  • ✅ keep this PR as an arm A data point (rejected: drop the note)
  • Pullfrog's finding on src/cli/models.ts:87: an arm's entry now drops every other entry of the same model, not only the identical one, since a same-model fallback hits the same quota. M B's list is [opus high, sol high].

Follow-ups

None.

Models

  • Size: M
  • Implementer: Claude Code, claude-opus-5-5, medium effort (arm A)
  • Implementer: Claude Code, claude-opus-5-5, low effort (iterate, S, arm A)

🤖 Generated with Claude Code

(Written by Claude Opus 5.5 with medium)

Review

mechanical fix by codex, gpt-6.1-sol, medium effort: Merged base and mechanically re·solved README.md, src/cli/main.ts, src/cli/models.ts, and src/cli/usage.ts conflicts, preserving writer A/B and risk reporting; ran nix develop --command just install, nix develop --command bunx prettier --write README.md src/cli/main.ts src/cli/models.ts src/cli/usage.ts, nix develop --command just check, and nix develop --command just test. (head 60b01c4)

Rewritten by quest review after each round.

  • Head: c39181b

  • Base: origin/main at d0f289d

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1d768-5891-7462-a0d4-d78f07b3ec1f, round 2

  • Score: 10/10, The routing fix resolves the earlier finding, and the incremental changes introduce no new findings.

  • Rounds: 2; findings 1 (medium 1): fixed 1, refuted 0, open 0

  • Findings: none

  • The last round's findings, at 607278e:

    • Fixed: medium, src/cli/models.ts:85: Preserve the effective writer size for armed iterates; src/cli/models.ts:81-82 now returns arm: null and the next size's writer list for Maintainer waits, so every assigned arm uses the PR's own size.
  • Reviewer: Codex, gpt-6.1-sol, medium effort, shadow

  • Session: 01a1d768-5891-76e3-b5c8-0d43566c30da

  • Rounds: 1; findings 0: fixed 0, refuted 0, open 0

9abb268 applies Pullfrog's suggested patch on src/cli/models.ts:87 unchanged on top of the reviewed head c39181b, which /quest-merge exempts from a new round.

kixelated and others added 2 commits November 13, 2026 16:06
quest unblocked --json names each ready quest's arm, picked by a hash of
its path, and opens models with that arm's entry; quest models --arm
prints an arm's entry for an iterate; Implementer lines record the arm as
(arm B); and quest usage rolls up outcomes and spend per size and arm.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Split each size's writers between two arms feat(models): split XS, S, M, and XL writers between two A/B arms Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 14, 2026 00:19
@coderabbitai

coderabbitai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a92c5a89-8cd8-4c8b-838f-811b3f067d2d
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — one minor suggestion inline.

Reviewed changes

This review covers the writer A/B split end to end: arm assignment, the quest models --arm surface, Implementer: parsing, the quest usage arm rollup, and the guide and skill edits.

  • Arm assignment — questWriters hashes the repository-relative quest path to pick arm A or B. It puts the arm's entry first and drops arms for L and for quests routed up a size by a Maintainer: wait.
  • quest models --arm — prints one arm's entry at the PR's size. It refuses to run together with --writer and refuses L.
  • Implementer: parsing — reads arm A or arm B from the line's notes, and Fable 5.2 is now a known model.
  • Usage rollup — the latest-branch tally logic moves into a shared outcomes(). armRows counts a PR only when it is sized, every line is readable, and every line names the same arm, and it charges every session's spend to that arm.
  • Guide and skills — iterate dispatch uses --arm for an armed PR, and /quest-run and /quest-iterate record the arm.
  • Quest tree — deletes writer-ab.md and points its links at this PR.

bun run check passes, and the cli, usage, and review suites pass under bun run test.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Comment thread src/cli/models.ts Resolved
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pullfrog[bot]
pullfrog Bot previously approved these changes Nov 14, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

This review covers the one commit pushed since the prior Pullfrog review.

  • Fallback filter matches on model alone — questWriters in src/cli/models.ts now drops every entry of the arm's model from the size's list, not only the identical model and effort. M arm B's list is [opus high, sol high], and the earlier thread on this is resolved.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 14, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 14, 2026
@moq-bot

moq-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

quest land watch: the merge queue removed this pull request at 9abb268 because it conflicts with main (merge_conflict).

The land fixer works on it in the background, and the watcher re-enqueues it at the back of the queue once it is fixed or judged flaky, or hands it back here with what it needs.

(written by quest land watch)

@moq-bot

moq-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

quest land watch: armed auto-merge on this pull request at 60b01c4, so it joins the back of the merge queue once its required checks pass, with the land fixer's fix (codex, gpt-6.1-sol, medium effort): Merged base and mechanically re·solved README.md, src/cli/main.ts, src/cli/models.ts, and src/cli/usage.ts conflicts, preserving writer A/B and risk reporting; ran nix develop --command just install, nix develop --command bunx prettier --write README.md src/cli/main.ts src/cli/models.ts src/cli/usage.ts, nix develop --command just check, and nix develop --command just test..

(written by quest land watch)

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

This review covers the merge of main (#716, risk tiers) into the branch since the prior Pullfrog review. I checked its conflict resolution with git show --cc.

  • src/cli/models.ts — kept CLASSIFIERS next to Arm, ARM_NAMES, ARMS, and the armed questWriters without changing either side.
  • src/cli/main.ts — merged the ./models import so it includes both CLASSIFIERS and the arm exports. The quest usage help now lists the writer A/B rollup and then the risk tier rollup.
  • src/cli/usage.ts — kept ArmRow and TierRow together, so Usage has both arms and tiers. In the per-PR text, the risk tier and risk: line from main sit alongside the arm notes on implementer: lines.
  • README.md — combined the two rollup descriptions in the quest usage prose.

tsc --noEmit passes, and the cli, usage, and review suites pass under bun run test (140 tests).

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@moq-bot
moq-bot Bot added this pull request to the merge queue Nov 14, 2026
Merged via the queue into main with commit d86e927 Nov 14, 2026
20 checks passed
@moq-bot
moq-bot Bot deleted the quest/m0/routing-evidence/writer-ab branch November 14, 2026 01:19
kixelated added a commit that referenced this pull request Nov 14, 2026
Combine #720's writer A/B arm rows with per-cell cost, reviewer rollup and spent.split.
A sub-agent known to be in another repository no longer inherits its parent's PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant