diff --git a/README.md b/README.md index 0adbd45c..31f777c9 100644 --- a/README.md +++ b/README.md @@ -132,14 +132,15 @@ $ quest --root examples/export unblocked 1 quest/m0/export.md $ quest --root examples/export unblocked --json --limit 1 -[{"path":"quest/m0/export.md","dependents":1,"hosts":[],"models":[{"harness":"claude","model":"claude-opus-5-5","effort":"medium"},{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}]}] +[{"path":"quest/m0/export.md","dependents":1,"hosts":[],"arm":"B","models":[{"harness":"claude","model":"claude-sonnet-5-5","effort":"high"},{"harness":"claude","model":"claude-opus-5-5","effort":"medium"},{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}]}] ``` Each plain output line contains the count, a tab, and the path. On a terminal, a heading explains the count. `--json` prints an ordered array, including `[]` when no quests are ready, and gives each quest the `hosts` its `## Hosts` -section names (empty when it runs anywhere) and the ranked `models` that write -it, as [Route work by size](#route-work-by-size) describes; `--limit ` +section names (empty when it runs anywhere), its writer A/B `arm`, and the +ranked `models` that write it, as [Route work by size](#route-work-by-size) +describes; `--limit ` requires a positive integer. Invalid trees fail with diagnostics and no ranking. `quest-run` uses the ranking unless you explicitly select work in another order. @@ -1273,6 +1274,19 @@ to the next only while nothing is written: when the CLI is missing or logged out, or the agent stops at its usage limit with nothing beyond the claim commit. Once a model writes, the PR is its own. +While the writer A/B runs, XS, S, M, and XL quests split 50/50 between two +arms, to measure where published guidance disagrees: arm A runs the table's +writer, and arm B runs Claude Code `claude-opus-5-5` medium for XS, +`claude-sonnet-5-5` high for S, `claude-opus-5-5` high for M, and +`claude-fable-5-2` medium for XL. A hash of the quest's path picks the arm, +so a restart or respawn keeps it; `quest unblocked --json` names it as `arm`, +`A` or `B`, and puts its entry first in `models`. L runs no arm, nor does a +quest with a `Maintainer:` decision open, which runs the next size's list, so +an arm always runs at its PR's size; their `arm` is `null`. The agent +that runs that entry ends its `Implementer:` line with the arm, as +`Implementer: Claude Code, claude-opus-5-5, high effort (arm B)`; a fallback +names none, so its PR counts in neither arm. + A `/quest-iterate` stays on the PR's writer, the model its `## Models` record's first `Implementer:` line names, so the other model can always review it. The coordinator dispatching it sizes the work left on the PR, not @@ -1290,7 +1304,9 @@ claude claude-opus-5-5 medium `--writer` prints the size's entry for that model, or else its entry at the nearest size that lists it, the larger on a tie; a model no list names is an -error. `--json` prints an array of `{harness, model, effort}` objects. The +error. An iterate on a PR whose first `Implementer:` line names an arm runs +that arm's entry at the PR's size instead, as `quest models M --arm B` +prints, and records it as `(iterate, XS, arm B)`. `--json` prints an array of `{harness, model, effort}` objects. The iterate records the size it was given in its `Implementer:` line, as `Implementer: Claude Code, claude-opus-5-5, medium effort (iterate, XS)`, so the model check can audit the routing. `quest review` still reviews the @@ -1612,7 +1628,9 @@ findings measure an effort step rather than the writer. Since a branch's tallies PR whose record holds its latest tally, by the most rounds and then the newest, counts them, and a record written before tallies counts as unreviewed. An `Implementer:` line's own size, as -`(iterate, XS)`, counts over the PR's; each risk tier's PRs, how many merged, +`(iterate, XS)`, counts over the PR's; the writer A/B by size and arm, with +the same outcomes and every session's spend, counting a sized PR only when +each of its `Implementer:` lines names the same arm; each risk tier's PRs, how many merged, how many a reviewer raised, and how many a later PR's `## Caused by` names, which shows whether the tiers predict what breaks; and, over the merged PRs, each online reviewer's coverage, final heads reviewed, findings per PR reviewed, and diff --git a/assets/AGENTS.md b/assets/AGENTS.md index 9a0a3b5d..02e89ab5 100644 --- a/assets/AGENTS.md +++ b/assets/AGENTS.md @@ -215,7 +215,7 @@ Leave the worktree you are running in and any holding uncommitted work or refuse Run each agent at an entry of `quest`'s ranked lists unless the repository's `AGENTS.md` names other models: as a native sub-agent on this session's harness, else through `quest agent`. Size each `/quest-iterate` you dispatch as a quest, by the work left on its PR, and name that size in its brief. -Start it on the PR's writer, the model of its first `Implementer:` line, at the entry `quest models --writer ` prints, waiting out its usage limit so the other model can review; a PR whose writer no list names, such as `unknown` or a retired model, takes its size's first entry that can run. +Start it on the PR's writer, the model of its first `Implementer:` line, at the entry `quest models --writer ` prints, or `quest models --arm ` when that line names a writer A/B arm, as `(arm B)`, naming the arm in its brief, waiting out its usage limit so the other model can review; a PR whose writer no list names, such as `unknown` or a retired model, takes its size's first entry that can run. `quest agent start --harness --model --effort --permission-mode ` runs a headless agent on that CLI's own login; start it as a background command in its own isolated worktree, so the harness reports its exit like a sub-agent's. `--permission-mode` is Claude Code's permission mode, or Codex's sandbox mode. diff --git a/assets/skills/iterate.md b/assets/skills/iterate.md index 7d6a2d0a..96aaf74b 100644 --- a/assets/skills/iterate.md +++ b/assets/skills/iterate.md @@ -10,7 +10,7 @@ Take it with `quest iterate [token]` before acting, passing a hold token yo If it refuses, stop and report what it printed; taking over a hold is the user's choice. If you cannot push to the PR's branch, stop and report. When another worktree holds the PR's branch, work in a detached checkout of its head. -Your `Implementer:` line ends with the size your brief gives the work, as `Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS)`. +Your `Implementer:` line ends with the size your brief gives the work, and the arm it names, as `Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS, arm B)`. Read the PR, its linked issues and quest, the reviews and comments posted on it, and the surrounding code. `quest status ` reports its checks, reviews by head, and unresolved threads in one call. diff --git a/assets/skills/run.md b/assets/skills/run.md index cab56e29..deee9e2d 100644 --- a/assets/skills/run.md +++ b/assets/skills/run.md @@ -27,6 +27,7 @@ Skip a quest a match already implements; one that only touches the same files is Each agent runs `/quest-start`; its `quest start` refusing a quest claimed since is a skip, not a failure. Start each on the first entry of its `models` that can run. +When it has an `arm` and runs that first entry, its brief says to end its `Implementer:` line with the arm, as `(arm B)`. Fall to the next when `quest agent` reports its CLI missing or logged out, or the agent stops on a usage limit with nothing written beyond the claim commit, continuing in that checkout and claim PR. Once a model has written anything, the PR is its: keep its worktree and session id to resume after the reset. Skip a quest whose list runs out, and report each harness's outage once. diff --git a/quest/m0/routing-evidence/README.md b/quest/m0/routing-evidence/README.md index 650d210f..fd47dfea 100644 --- a/quest/m0/routing-evidence/README.md +++ b/quest/m0/routing-evidence/README.md @@ -42,7 +42,13 @@ Readout, the agent's choices for the implementer to revisit: - Start one week after both experiments are live. From `quest usage --json`, compute each arm's variance and the number of PRs each arm needs to detect a difference worth acting on. If the arms have not reached it, - rewrite the `Waiting` item below for the date they will, and stop. + rewrite the `Waiting` item below for the date they will, and stop. XL + quests are rare, so report how many each XL arm got rather than wait for + a detectable difference there (user, 2026-11-13). +- `quest usage`'s `arms` rollup counts each sized PR whose every + `Implementer:` line names the same arm, under its first line's writer, + with every session's spend on it. Escapes are too rare to separate the + arms (about 2% of PRs): report them alongside, not as the measure. - For each writer arm, compare first-round findings from the fixed efforts in `src/cli/models.ts`, review rounds, hand-backs, later `## Caused by` culprits, and cost per merged PR. For each shadow pair, compare findings the routed reviewer @@ -54,7 +60,9 @@ Readout, the agent's choices for the implementer to revisit: each finding's location from the branch's saved rounds under the clones' `.git/quest/review/`. Recommend each table change to the maintainer with its tally. End the writer A/B by deleting its arms from - `src/cli/models.ts`. Keep the shadow reviewer only if the maintainer + `src/cli/models.ts`, with `quest models --arm`, the `arm` field of + `quest unblocked --json`, and the arm notes in the README, guide, and + `/quest-run` and `/quest-iterate`. Keep the shadow reviewer only if the maintainer chooses to. - Carry the tallies into [the model check](/quest/m1/model-check.md)'s baseline. @@ -106,7 +114,7 @@ Baseline from the 2026-11-11 audit, at API list price: - moq-dev/quest: one week of PRs since both the shadow reviewer (`quest review`'s, PR titled "quest: run a shadow reviewer one effort below the routed one") and - [the writer A/B](/quest/m0/routing-evidence/writer-ab.md) merged; check + the writer A/B ([moq-dev/quest#720](https://github.com/moq-dev/quest/pull/720)) merged; check their merge dates with `gh pr list --state merged` and the day with `date` ## Related diff --git a/quest/m0/routing-evidence/charts.md b/quest/m0/routing-evidence/charts.md index 0f60984f..0bc684e3 100644 --- a/quest/m0/routing-evidence/charts.md +++ b/quest/m0/routing-evidence/charts.md @@ -46,7 +46,7 @@ Decided 2026-11-11 (user, /quest-plan in a /quest-run session, when picking The benchmarks behind the XS writer change to Sonnet 5.5 high ([moq-dev/quest#646](https://github.com/moq-dev/quest/pull/646)) and the XS -arm of the [writer A/B](/quest/m0/routing-evidence/writer-ab.md) +arm of the writer A/B ([moq-dev/quest#720](https://github.com/moq-dev/quest/pull/720)) ([moq-dev/quest#644](https://github.com/moq-dev/quest/pull/644)), both decided on 2026-11-11, for the first table: diff --git a/quest/m0/routing-evidence/writer-ab.md b/quest/m0/routing-evidence/writer-ab.md deleted file mode 100644 index 47365bde..00000000 --- a/quest/m0/routing-evidence/writer-ab.md +++ /dev/null @@ -1,77 +0,0 @@ -# [M] Split each size's writers between two arms - -## Goal - -`quest unblocked --json` routes each XS and S quest to Sonnet 5.5 high or -Opus 5.5 medium, each M quest to Opus medium or high, and each XL quest to -Opus xhigh or Fable 5.2 medium, 50/50. The split is stable per quest, -and the review efforts in `src/cli/models.ts` stay fixed while it runs. Each `Implementer:` line -records its arm, iterates stay on the PR's arm, and -`quest usage` reports cost, rounds, findings, hand-backs, and later -culprits per arm. - -## Plan - -Decided 2026-11-11 (user, /quest-plan, -[routing evidence](/quest/m0/routing-evidence/README.md)): - -- Which cells? - - ✅ S Opus low against medium, and M Opus medium against high - - Rejected: XS Haiku xhigh against Opus low, since XS PRs rarely draw - findings and the signal would be weak -- Amended 2026-11-11 (user, /quest-run): XS writers move from Haiku - xhigh to Sonnet 5.5 high in a separate routing PR, which also updates - `src/cli/models.ts`, the README, and the model check's XS baseline. Two - Haiku agents in a row ran out of context, since the maintainer's Claude - Code settings cap `claude-haiku-5-5`'s `autoCompactWindow` at 100k to - stay under Haiku's 5x price past 100k input. Does XS get an arm? - - ✅ Sonnet high against Opus low: published results disagree on this - pair at about the same cost. Anthropic's - [cost guide](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence) - has them even on its SWE-bench Pro subset (87.6% and 87.4%, about - $0.11 per solved task), while - [Artificial Analysis](https://artificialanalysis.ai/models/comparisons/claude-sonnet-5-5-high-vs-claude-opus-5-5-low) - has Sonnet high 12.6 points ahead on Terminal-Bench 4.0 and Opus low - about 40% cheaper per task. XS PRs rarely draw findings, so cost, - rounds, and hand-backs carry most of its signal. - - Rejected: Sonnet medium against Sonnet high, since the same cost - guide puts Sonnet medium about 7 points below high on its SWE-bench - Pro subset and 15 below on its internal agentic-coding set; no XS arm -- Re-armed 2026-11-13 (user), to test where Anthropic's - [model guide](https://claude.dev/blog/which-claude-model-should-you-build-with/) - and Theo Browne's - [Opus against Fable video](https://www.youtube.com/watch?v=XtykeVoSmP8) - disagree: - - S Sonnet high against Opus medium: the guide puts scoped coding on - Sonnet, Theo finds Sonnet runs up output tokens and costs about what - Opus does. S's table entry moved from Opus low to Opus medium, the - guide's default, since both call Opus low too weak. - - XS Sonnet high against Opus medium rather than Opus low, the same - contrast as S (agent's choice, since no table entry runs Opus low any - more). - - M Opus medium against high, as before: Theo finds medium ends its turn - too early and calls high the best effort. - - XL Opus xhigh against Fable medium: the guide moves up to Fable rather - than past high, while Theo finds Opus checks verifiable work more - carefully and a subscription gives about a quarter as much Fable. XL's - table entry stays Opus xhigh for that subscription cap (rejected: - Fable medium; dropping the Fable arm). XL quests are rare, so report - how many each arm got rather than wait for a detectable difference. -- Escapes are too rare to separate the arms, about 2% of PRs. Compare - first-round findings under the fixed review efforts, rounds, hand-backs, - and cost, and report escapes alongside. - -Agent's choices, for the implementer to revisit: - -- Pick the arm from a hash of the quest's path, so restarting or - respawning a quest keeps its arm, and a dispatcher reads it from - `quest unblocked --json` rather than drawing again. -- Record it on the Implementer line beside the size an iterate already - records, such as `(arm B)` or `(iterate, S, arm B)`, and extend - `implementer()` in `src/cli/models-record.ts` to parse both, since today it - accepts only `(XS)` and `(iterate, XS)`. A writer that ran off its arm, - such as a fallback to Sol, counts in neither arm. -- `quest models` takes only a size, so an iterate's entry comes from the - arm on the PR's first Implementer line: `--writer` also takes that line's - effort, or a new option names the arm. -- [The readout](/quest/m0/routing-evidence/README.md) ends the split. diff --git a/quest/m1/model-check.md b/quest/m1/model-check.md index aa234994..bf3e24ef 100644 --- a/quest/m1/model-check.md +++ b/quest/m1/model-check.md @@ -123,7 +123,7 @@ checkpoints and outages to count by hand: [video](https://www.youtube.com/watch?v=XtykeVoSmP8) estimates about $3k a month of Fable against $14k of Opus on the $200 plan). XS stays Sonnet high, since `quest usage` had no Sonnet medium runs to compare. The - [writer A/B](/quest/m0/routing-evidence/writer-ab.md) tests each change + writer A/B ([moq-dev/quest#720](https://github.com/moq-dev/quest/pull/720)) tests each change against its alternative, Fable medium on XL included; tally the S cell fresh. - Compare both tables' models (in `src/cli/models.ts`) with each vendor's current models: the newest diff --git a/src/cli/main.ts b/src/cli/main.ts index 26071feb..d76ce9a9 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -50,7 +50,17 @@ import { watch, type WatchOptions } from "./land-watch"; import { landed, landingCheck } from "./landed"; import { type Report, fixHeadings, fixLinks, moveQuest, removeQuest } from "./move"; import { type ResolveOptions, resolveFile } from "./resolve"; -import { CLASSIFIERS, DEFAULT_SIZE, REVIEWERS, WRITERS, questWriters, writerAt } from "./models"; +import { + ARMS, + ARM_NAMES, + type Arm, + CLASSIFIERS, + DEFAULT_SIZE, + REVIEWERS, + WRITERS, + questWriters, + writerAt, +} from "./models"; import { ReviewInputError, ReviewLimitError, @@ -304,10 +314,12 @@ export function main(argv: string[], out: Output): number | Promise { "section or a ## Scheduled date after today's local date. Counts exclude structural epic membership " + "but include the quests an epic's ## Required holds back. Other prerequisites may remain for dependents. " + "Invalid trees fail before printing a ranking. Prints a count and path per line; --json prints an " + - "ordered array of {path, dependents, hosts, models} objects, where hosts lists the operating systems " + + "ordered array of {path, dependents, hosts, arm, models} objects, where hosts lists the operating systems " + "(linux, macos, windows) under the quest's ## Hosts, empty when it runs anywhere, and models ranks the {harness, model, effort} " + "entries that write the quest: its size's list, or the next size's while a Maintainer: item under its " + - "## Waiting leaves a decision open (quest models lists them).\n\n" + + "## Waiting leaves a decision open (quest models lists them). While the writer A/B runs, arm is A or B for " + + "a size that runs one, and null for L or a quest routed up for an open Maintainer: item, and models opens with that arm's entry; the agent that runs it " + + "records the arm on its Implementer line.\n\n" + "--unclaimed ranks the fetched upstream default branch's tree instead of the checkout, then drops " + "quests someone holds: a quest branch upstream, or an open PR from that branch in any repository. A " + "branch whose tip is a merged PR's head holds nothing; a closed, unmerged PR's branch still does. A " + @@ -348,7 +360,7 @@ export function main(argv: string[], out: Output): number | Promise { JSON.stringify( shown.map((quest) => { const doc = byPath.get(quest.path)!; - return { ...quest, hosts: hosts(doc), models: questWriters(doc) }; + return { ...quest, hosts: hosts(doc), ...questWriters(doc) }; }), ), ); @@ -1097,15 +1109,31 @@ export function main(argv: string[], out: Output): number | Promise { "--writer prints the one entry that runs that model on the work instead, for an iterate, which stays " + "on its PR's writer: the size's own entry for the model, else its entry at the nearest size that " + "lists it, the larger on a tie, so an XS iterate on an Opus PR runs Opus at S's effort. A model no " + - "list names is an error. Prints harness, model, and effort per line, tab-separated; --json prints an " + + "list names is an error. While the writer A/B runs, each armed size's quests split by a hash of their " + + "path between arm A, the size's first entry, and arm B: " + + Object.entries(ARMS) + .map(([size, arms]) => `${size} B ${arms.B.harness} ${arms.B.model} ${arms.B.effort}`) + .join("; ") + + ". --arm prints that arm's entry at the PR's size instead, for an iterate on a PR whose first " + + "Implementer line names the arm. Prints harness, model, and effort per line, tab-separated; --json prints an " + "array of {harness, model, effort} objects.", ) .addArgument(new Argument("", "The work's size").choices(SIZES)) .option("--writer ", "Print only the entry that runs this model, the PR's first Implementer") + .addOption( + new Option("--arm ", "Print only this writer A/B arm's entry, the PR's first Implementer's").choices( + ARM_NAMES, + ), + ) .option("--json", "Print a JSON array") - .action((size: Size, options: { writer?: string; json?: boolean }) => { + .action((size: Size, options: { writer?: string; arm?: Arm; json?: boolean }) => { let entries = WRITERS[size]; - if (options.writer !== undefined) { + if (options.arm !== undefined) { + if (options.writer !== undefined) throw new UsageError("give --writer or --arm, not both"); + const arms = ARMS[size]; + if (!arms) throw new UsageError(`${size} runs no writer arms`); + entries = [arms[options.arm]]; + } else if (options.writer !== undefined) { const entry = writerAt(size, options.writer); if (entry === null) throw new UsageError(`no write list names ${options.writer}`); entries = [entry]; @@ -1136,7 +1164,8 @@ export function main(argv: string[], out: Output): number | Promise { "review thread it opens or a [P] line in its review body, and its first answer's first word takes or " + "declines it. Rollups follow: spend by size, role, harness, model, and " + "effort, accuracy by size and implementer, summing each branch's tallies but the shadow's once, where an Implementer line's own size, as (iterate, XS), wins " + - "over the PR's, each risk tier's PRs, merged PRs, tiers a reviewer raised, and PRs a later one fixed, and each online reviewer's coverage and findings over the merged PRs. Without token-max on PATH, it says how to install it and reports the PRs' records alone. " + + "over the PR's, the writer A/B by size and arm, counting a sized PR whose every Implementer line names the same arm, as (arm B), " + + "under its first line's writer with the spend of every session on it, each risk tier's PRs, merged PRs, tiers a reviewer raised, and PRs a later one fixed, and each online reviewer's coverage and findings over the merged PRs. Without token-max on PATH, it says how to install it and reports the PRs' records alone. " + "--json prints the same report as one object.\n\n" + "Uses GH_TOKEN, GITHUB_TOKEN, or gh auth token. Exits 0 when done, 2 for invalid arguments, and 1 for " + "other failures.", diff --git a/src/cli/models-record.ts b/src/cli/models-record.ts index 3a31b9d9..04b67358 100644 --- a/src/cli/models-record.ts +++ b/src/cli/models-record.ts @@ -1,7 +1,7 @@ // A pull request body's `## Models` record, which names the PR's size and each session that implemented it. import { SIZES } from "../core"; -import { REVIEWERS, WRITERS } from "./models"; +import { ARMS, ARM_NAMES, type Arm, REVIEWERS, WRITERS } from "./models"; const SECTION = /^## Models[ \t]*\r?$/m; @@ -21,6 +21,8 @@ export interface Implementer { effort: string; /** The size its own line ends with, as `(iterate, XS)`. */ size: string | null; + /** The writer A/B arm its line ends with, as `(arm B)` or `(iterate, S, arm B)`. */ + arm: Arm | null; } /** @@ -55,7 +57,9 @@ const key = (name: string) => .replace(/[\s.]+/g, "-") .replace(/^claude-/, ""); const KNOWN = new Map( - [...Object.values(WRITERS), ...Object.values(REVIEWERS)].flat().map(({ model }) => [key(model), model]), + [...Object.values(WRITERS), ...Object.values(REVIEWERS), ...Object.values(ARMS).flatMap(Object.values)] + .flat() + .map(({ model }) => [key(model), model]), ); /** @@ -72,8 +76,8 @@ function modelId(name: string, harness: string, loose = false): string | null { /** * The fields of an `Implementer: , , effort` item, as agents write it: the model by ID or - * display name, the harness or effort left out, the effort bare or `effort not reported`, and a size among the - * notes of a parenthetical it ends with, as `(iterate, XS)`. Null when it names no model or has more fields; a lone + * display name, the harness or effort left out, the effort bare or `effort not reported`, and a size and writer A/B + * arm among the notes of a parenthetical it ends with, as `(iterate, XS, arm B)`. Null when it names no model or has more fields; a lone * field reads only as a listed model's ID or display name, or as an ID. */ export function implementer(item: string): Implementer | null { @@ -82,11 +86,13 @@ export function implementer(item: string): Implementer | null { if (notes) rest = rest.slice(0, notes.index); const parts = rest.split(/,\s*/).map((part) => part.trim()); if (parts.length > 3) return null; - const size = notes?.[1]!.split(/,\s*/).find((note) => (SIZES as readonly string[]).includes(note)) ?? null; + const listed = notes?.[1]!.split(/,\s*/) ?? []; + const size = listed.find((note) => (SIZES as readonly string[]).includes(note)) ?? null; + const arm = ARM_NAMES.find((name) => listed.includes(`arm ${name}`)) ?? null; // A lone field is a model only as a listed name or an ID, since `unknown` or another name alone may be a harness. if (parts.length === 1) { const id = modelId(parts[0]!, ""); - return id !== null && id !== "unknown" ? { harness: "unknown", model: id, effort: "unknown", size } : null; + return id !== null && id !== "unknown" ? { harness: "unknown", model: id, effort: "unknown", size, arm } : null; } // The model is the last field, or the one before the effort; a second field `unknown` is the effort, since a // two-field line names its model. @@ -104,5 +110,5 @@ export function implementer(item: string): Implementer | null { if (!named) return null; effort = named[1]!; } - return { harness, model, effort, size }; + return { harness, model, effort, size, arm }; } diff --git a/src/cli/models.ts b/src/cli/models.ts index fa4c3ac7..8230e78d 100644 --- a/src/cli/models.ts +++ b/src/cli/models.ts @@ -2,6 +2,7 @@ // every repository; a repository's AGENTS.md may override them in prose, which // the session following it passes as flags. The model check // (quest/m1/model-check.md) audits these lists each milestone. +import { createHash } from "node:crypto"; import { type Doc, SIZES, type Size, onMaintainer, size, waits } from "../core"; import type { HarnessName } from "./agent"; @@ -18,6 +19,7 @@ export const DEFAULT_SIZE: Size = "M"; const sol = (effort: string): Model => ({ harness: "codex", model: "gpt-6.1-sol", effort }); const opus = (effort: string): Model => ({ harness: "claude", model: "claude-opus-5-5", effort }); const sonnet = (effort: string): Model => ({ harness: "claude", model: "claude-sonnet-5-5", effort }); +const fable = (effort: string): Model => ({ harness: "claude", model: "claude-fable-5-2", effort }); /** * Each size's writers, ranked: a spawn starts the quest on the first that can @@ -49,6 +51,10 @@ export const REVIEWERS: Record = { XL: [sol("xhigh"), opus("xhigh")], }; +/** One side of the writer A/B: A runs the size's own first writer, B the alternative. */ +export type Arm = "A" | "B"; +export const ARM_NAMES: readonly Arm[] = ["A", "B"]; + /** * The models that set a PR's risk tier from its diff, ranked: the first that * can run classifies, past any logged out or at its usage limit. The tier never depends on who wrote the change, @@ -62,15 +68,36 @@ export const CLASSIFIERS: readonly Model[] = [ ]; /** - * The writers that start quest `doc`: its size's list, or the next size's - * while a `Maintainer:` wait leaves it a decision to settle. + * The writer A/B: each armed size's quests split between its first writer + * and an alternative where published results disagree, until the readout in + * quest/m0/routing-evidence/README.md deletes it. L runs no arm. + */ +export const ARMS: Partial>> = { + XS: { A: sonnet("high"), B: opus("medium") }, + S: { A: opus("medium"), B: sonnet("high") }, + M: { A: opus("medium"), B: opus("high") }, + XL: { A: opus("xhigh"), B: fable("medium") }, +}; + +/** + * The writers that start quest `doc`, ranked, and its writer A/B arm. A quest + * whose size runs an arm starts on the arm's entry, then the size's list + * without it; the arm hashes the quest's path, so a restart or respawn draws + * the same one. A `Maintainer:` wait leaves it a decision to settle, so it + * runs the next size's list and no arm, which keeps each arm at its PR's size. */ -export function questWriters(doc: Doc): readonly Model[] { +export function questWriters(doc: Doc): { arm: Arm | null; models: readonly Model[] } { const sized = size(doc); // `quest check` requires a size of every quest that is not an epic, and only those can be ready. if (sized === null) throw new Error(`${doc.path} has no size`); - const bump = waits(doc).some(onMaintainer) ? 1 : 0; - return WRITERS[SIZES[Math.min(SIZES.indexOf(sized) + bump, SIZES.length - 1)]]; + if (waits(doc).some(onMaintainer)) + return { arm: null, models: WRITERS[SIZES[Math.min(SIZES.indexOf(sized) + 1, SIZES.length - 1)]!] }; + const arms = ARMS[sized]; + if (!arms) return { arm: null, models: WRITERS[sized] }; + const arm = ARM_NAMES[createHash("sha256").update(doc.path).digest()[0]! & 1]!; + const entry = arms[arm]; + const same = (e: Model) => e.model === entry.model; + return { arm, models: [entry, ...WRITERS[sized].filter((e) => !same(e))] }; } /** diff --git a/src/cli/usage.ts b/src/cli/usage.ts index f33491e3..52819911 100644 --- a/src/cli/usage.ts +++ b/src/cli/usage.ts @@ -3,6 +3,7 @@ // named; quest ties each to a PR by the branches and repositories token-max exports for it. import { spawnSync } from "node:child_process"; import { type RecordedRisk, type Size, SIZES, TIERS, type Tier, handedBack, questBranch, recordedRisk } from "../core"; +import type { Arm } from "./models"; import { recorded } from "./models-record"; import { type ClaimDependencies, upstream } from "./claims"; import { repositoryPath } from "../github/client"; @@ -75,7 +76,7 @@ export interface PullRow { /** Its body has a `## Waiting` section, so it was handed back to the maintainer. */ handedBack: boolean; size: Size | null; - implementers: { harness: string; model: string; effort: string; size: Size | null }[]; + implementers: { harness: string; model: string; effort: string; size: Size | null; arm: Arm | null }[]; /** The `Implementer:` items it cannot read, which count toward no implementer. */ unread: string[]; /** Each reviewer the PR's `## Review` record names, with its latest round and its tally. */ @@ -122,6 +123,15 @@ export interface AccuracyRow { fixes: number; } +/** One writer A/B arm at one size, over the PRs every Implementer line of which names it. */ +export interface ArmRow extends Omit { + size: Size; + arm: Arm; + /** The sessions on its PRs in every role, and their spend. */ + sessions: number; + spend: Tokens; +} + /** The PRs of one risk tier, or of none (`tier` null): how many merged, how many a reviewer raised, and later fixed. */ export interface TierRow { tier: Tier | null; @@ -156,6 +166,8 @@ export interface Usage { pulls: PullRow[]; spend: SpendRow[]; accuracy: AccuracyRow[]; + /** The writer A/B's arms, while it runs. */ + arms: ArmRow[]; tiers: TierRow[]; online: OnlineRow[]; /** The sessions on no PR reported here, from every repository and agent token-max reads; null when narrowed. */ @@ -538,12 +550,31 @@ function spendRows(pulls: PullRow[]): SpendRow[] { ); } -function accuracyRows(pulls: PullRow[]): AccuracyRow[] { - const rows = new Map(); - // A branch's saved tallies span every PR from it, so only one PR that has both a tally and an implementer - // to count it for counts them: the one whose record holds the latest tally, as the most rounds show, since - // each round rewrites only its own PR's record; the newest breaks a tie. A deleted fork has no repository, - // so its PR stands alone. +/** The fields an accuracy row and an arm row both tally over their PRs. */ +type Outcomes = Pick< + AccuracyRow, + "pulls" | "merged" | "handedBack" | "reviewed" | "scored" | "score" | "tally" | "landFixes" | "fixes" +>; + +const noOutcomes = (): Outcomes => ({ + pulls: 0, + merged: 0, + handedBack: 0, + reviewed: 0, + scored: 0, + score: 0, + tally: { rounds: 0, findings: { high: 0, medium: 0, low: 0 }, fixed: 0, refuted: 0, settled: 0, open: 0 }, + landFixes: 0, + fixes: 0, +}); + +/** + * Adds a PR's outcomes to rollup rows. A branch's saved tallies span every PR from it, so only one PR that has both + * a tally and an implementer to count it for counts them: the one whose record holds the latest tally, as the most + * rounds show, since each round rewrites only its own PR's record; the newest breaks a tie. A deleted fork has no + * repository, so its PR stands alone. + */ +function outcomes(pulls: PullRow[]): (row: Outcomes, each: PullRow) => void { const branch = (each: PullRow) => JSON.stringify([each.repository ?? each.number, each.branch]); const rounds = (each: PullRow) => each.review.reduce((sum, review) => sum + (review.shadow ? 0 : (review.tally?.rounds ?? 0)), 0); @@ -558,68 +589,54 @@ function accuracyRows(pulls: PullRow[]): AccuracyRow[] { ) latest.set(branch(each), each); } + return (row, each) => { + row.pulls++; + if (each.state === "merged") row.merged++; + if (each.handedBack) row.handedBack++; + row.fixes += each.fixedBy.length; + row.landFixes += each.landFixes.filter((by) => by !== MERGER).length; + // The shadow's findings measure an effort step, not the writer, so they stay out of the writer's row. + const tallied = + latest.get(branch(each)) === each + ? each.review.filter((review) => review.tally !== null && !review.shadow) + : []; + if (tallied.length > 0) { + row.reviewed++; + row.tally.rounds += Math.max(...tallied.map((review) => review.tally!.rounds)); + } + for (const { score, tally } of tallied) { + if (score !== null) { + row.scored++; + row.score += score; + } + for (const severity of SEVERITIES) row.tally.findings[severity] += tally!.findings[severity]; + row.tally.fixed += tally!.fixed; + row.tally.refuted += tally!.refuted; + row.tally.settled += tally!.settled; + row.tally.open += tally!.open; + } + }; +} + +function accuracyRows(pulls: PullRow[]): AccuracyRow[] { + const rows = new Map(); + const count = outcomes(pulls); for (const each of pulls) { // A session that pushed twice is one implementer of the PR, whether or not each line names its size. const distinct = new Map( - each.implementers.map((i) => { - const at = { ...i, size: i.size ?? each.size }; + each.implementers.map(({ harness, model, effort, size }) => { + const at = { harness, model, effort, size: size ?? each.size }; return [JSON.stringify(at), at]; }), ); for (const implementer of distinct.values()) { - const size = implementer.size; - const id = JSON.stringify([size, implementer.harness, implementer.model, implementer.effort]); + const id = JSON.stringify([implementer.size, implementer.harness, implementer.model, implementer.effort]); let row = rows.get(id); if (!row) { - row = { - size, - harness: implementer.harness, - model: implementer.model, - effort: implementer.effort, - pulls: 0, - merged: 0, - handedBack: 0, - reviewed: 0, - scored: 0, - score: 0, - tally: { - rounds: 0, - findings: { high: 0, medium: 0, low: 0 }, - fixed: 0, - refuted: 0, - settled: 0, - open: 0, - }, - landFixes: 0, - fixes: 0, - }; + row = { ...implementer, ...noOutcomes() }; rows.set(id, row); } - row.pulls++; - if (each.state === "merged") row.merged++; - if (each.handedBack) row.handedBack++; - row.fixes += each.fixedBy.length; - row.landFixes += each.landFixes.filter((by) => by !== MERGER).length; - // The shadow's findings measure an effort step, not the writer, so they stay out of the writer's row. - const tallied = - latest.get(branch(each)) === each - ? each.review.filter((review) => review.tally !== null && !review.shadow) - : []; - if (tallied.length > 0) { - row.reviewed++; - row.tally.rounds += Math.max(...tallied.map((review) => review.tally!.rounds)); - } - for (const { score, tally } of tallied) { - if (score !== null) { - row.scored++; - row.score += score; - } - for (const severity of SEVERITIES) row.tally.findings[severity] += tally!.findings[severity]; - row.tally.fixed += tally!.fixed; - row.tally.refuted += tally!.refuted; - row.tally.settled += tally!.settled; - row.tally.open += tally!.open; - } + count(row, each); } } return [...rows.values()].sort((a, b) => @@ -627,6 +644,36 @@ function accuracyRows(pulls: PullRow[]): AccuracyRow[] { ); } +/** + * The writer A/B's rollup: each sized PR whose every Implementer line names the same arm counts in it, under the + * size and writer its first line names, with every session's spend on it; a writer off its arm, such as a fallback + * that names none, leaves its PR in neither. + */ +function armRows(pulls: PullRow[]): ArmRow[] { + const rows = new Map(); + const count = outcomes(pulls); + for (const each of pulls) { + const first = each.implementers[0]; + if (!first?.arm || each.size === null || each.unread.length > 0) continue; + if (each.implementers.some((implementer) => implementer.arm !== first.arm)) continue; + const { harness, model, effort, arm } = first; + const id = JSON.stringify([each.size, arm, harness, model, effort]); + let row = rows.get(id); + if (!row) { + row = { size: each.size, arm, harness, model, effort, ...noOutcomes(), sessions: 0, spend: zero() }; + rows.set(id, row); + } + count(row, each); + for (const session of each.sessions) { + row.sessions++; + for (const spent of session.models) add(row.spend, spent); + } + } + return [...rows.values()].sort((a, b) => + order([a.size, a.arm, a.harness, a.model, a.effort], [b.size, b.arm, b.harness, b.model, b.effort]), + ); +} + /** One row per risk tier among `pulls`, lowest first, and PRs with none last. */ function tierRows(pulls: PullRow[]): TierRow[] { return [...TIERS, null].flatMap((tier) => { @@ -745,6 +792,7 @@ export async function usage(root: string, options: UsageOptions, deps: UsageDepe pulls: rows, spend: spendRows(rows), accuracy: accuracyRows(rows), + arms: armRows(rows), tiers: tierRows(rows), online: onlineRows(rows), unattributed, @@ -799,8 +847,10 @@ export function usageLines(report: Usage): string[] { lines.push( ` risk: ${row.risk.tier}, ${row.risk.reason}${row.risk.tier === row.risk.classified ? "" : `; raised from ${row.risk.classified} by a reviewer`}`, ); - for (const i of row.implementers) - lines.push(` implementer: ${i.harness}, ${i.model}, ${i.effort} effort${i.size ? ` (${i.size})` : ""}`); + for (const i of row.implementers) { + const notes = [i.size, i.arm && `arm ${i.arm}`].filter(Boolean).join(", "); + lines.push(` implementer: ${i.harness}, ${i.model}, ${i.effort} effort${notes ? ` (${notes})` : ""}`); + } for (const item of row.unread) lines.push(` unread: ${item}`); for (const r of row.review) lines.push( @@ -855,6 +905,22 @@ export function usageLines(report: Usage): string[] { ]), ), ); + if (report.arms.length) + lines.push( + "", + "Writer A/B by size and arm:", + ...table( + report.arms.map((a) => [ + " ", + a.size, + a.arm, + a.harness, + a.model, + a.effort, + `${a.pulls} PRs, ${a.merged} merged, ${a.handedBack} handed back, ${a.reviewed} reviewed: rounds ${mean(a.tally.rounds, a.reviewed)}, ${findingsText(a.tally)}; ${a.fixes} later fixes; ${a.sessions} sessions, ${tokens(a.spend)}`, + ]), + ), + ); if (report.tiers.length) lines.push( "", diff --git a/test/cli.test.ts b/test/cli.test.ts index 47d6afe5..049ae8e5 100644 --- a/test/cli.test.ts +++ b/test/cli.test.ts @@ -151,11 +151,13 @@ test("a document that is not UTF-8 is an error", () => { test("unblocked prints ranked counts and paths, with deterministic JSON and a limit", () => { const tree = Tree.baseline(); expect(tree.run("unblocked")).toEqual({ code: 0, stdout: "1\tquest/m0/epic/one.md\n", stderr: "" }); + const sonnetHigh = '{"harness":"claude","model":"claude-sonnet-5-5","effort":"high"}'; const opusMedium = '{"harness":"claude","model":"claude-opus-5-5","effort":"medium"}'; const solMedium = '{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}'; + // Its path hashes to arm B, which S runs on Sonnet before the size's own list. expect(tree.run("unblocked", "--json", "--limit", "1")).toEqual({ code: 0, - stdout: `[{"path":"quest/m0/epic/one.md","dependents":1,"hosts":[],"models":[${opusMedium},${solMedium}]}]\n`, + stdout: `[{"path":"quest/m0/epic/one.md","dependents":1,"hosts":[],"arm":"B","models":[${sonnetHigh},${opusMedium},${solMedium}]}]\n`, stderr: "", }); tree.write("quest/m0/epic/two.md", "# [S] Two\n\n## Goal\n\nWork.\n"); @@ -170,15 +172,19 @@ test("unblocked reports the hosts a quest names", () => { test("unblocked routes a quest with a decision open one size up, and XL stays XL", () => { const models = (tree: Tree) => - JSON.parse(tree.run("unblocked", "--json").stdout).map((quest: { models: { effort: string }[] }) => - quest.models.map((entry) => entry.effort).join(" "), + JSON.parse(tree.run("unblocked", "--json").stdout).map( + (quest: { arm: string | null; models: { effort: string }[] }) => + [String(quest.arm), ...quest.models.map((entry) => entry.effort)].join(" "), ); + // Its path hashes to arm B at every size that runs one; a decision left open runs no arm. const tree = Tree.baseline().append("quest/m0/epic/one.md", "\n## Waiting\n\n- moq-dev/moq: a release.\n"); - expect(models(tree)).toEqual(["medium medium"]); + expect(models(tree)).toEqual(["B high medium medium"]); tree.append("quest/m0/epic/one.md", "- Maintainer: pick a name.\n"); - expect(models(tree)).toEqual(["medium high"]); + expect(models(tree)).toEqual(["null medium high"]); tree.write("quest/m0/epic/one.md", "# [XL] One\n\n## Goal\n\nWork.\n\n## Waiting\n\n- Maintainer: pick.\n"); - expect(models(tree)).toEqual(["xhigh"]); + expect(models(tree)).toEqual(["null xhigh"]); + tree.write("quest/m0/epic/one.md", "# [XL] One\n\n## Goal\n\nWork.\n"); + expect(models(tree)).toEqual(["B medium xhigh"]); }); test("models ranks a size's writers, or the one entry that runs a PR's writer", () => { @@ -201,6 +207,33 @@ test("models ranks a size's writers, or the one entry that runs a PR's writer", expect(unknown.code).toBe(2); expect(unknown.stderr).toContain("no write list names unknown"); expect(tree.run("models", "XXL").code).toBe(2); + const arm = (size: string, name: string) => tree.run("models", size, "--arm", name); + expect(arm("M", "B").stdout).toBe("claude\tclaude-opus-5-5\thigh\n"); + expect(arm("XL", "B").stdout).toBe("claude\tclaude-fable-5-2\tmedium\n"); + expect(arm("XS", "A").stdout).toBe("claude\tclaude-sonnet-5-5\thigh\n"); + expect(arm("L", "A")).toMatchObject({ code: 2, stderr: expect.stringContaining("L runs no writer arms") }); + expect(arm("M", "C").code).toBe(2); + expect(tree.run("models", "M", "--arm", "A", "--writer", "claude-opus-5-5").code).toBe(2); +}); + +test("unblocked splits a size's quests between its arms by path, and keeps each quest's arm", () => { + const tree = Tree.baseline(); + for (let n = 0; n < 40; n++) tree.write(`quest/m0/epic/q${n}.md`, `# [M] Q${n}\n\n## Goal\n\nWork.\n`); + const arms = () => + Object.fromEntries( + JSON.parse(tree.run("unblocked", "--json").stdout).map((quest: { path: string; arm: string }) => [ + quest.path, + quest.arm, + ]), + ); + const first = arms(); + const counts = Object.values(first).reduce>( + (sum, name) => ({ ...sum, [String(name)]: (sum[String(name)] ?? 0) + 1 }), + {}, + ); + // The 40 new quests, and one.md's B. + expect(counts).toEqual({ A: 21, B: 20 }); + expect(arms()).toEqual(first); }); test("unblocked explains the count on a terminal, including NO_COLOR", () => { diff --git a/test/review.test.ts b/test/review.test.ts index 1fc622f3..dc90d154 100644 --- a/test/review.test.ts +++ b/test/review.test.ts @@ -1853,11 +1853,11 @@ test("a Models record reads its size and every implementing model", () => { implementers: [ { item: "Implementer: Claude Code, claude-opus-5-5, high effort", - fields: { harness: "Claude Code", model: "claude-opus-5-5", effort: "high", size: null }, + fields: { harness: "Claude Code", model: "claude-opus-5-5", effort: "high", size: null, arm: null }, }, { item: "Implementer: Codex, unknown, unknown effort", - fields: { harness: "Codex", model: "unknown", effort: "unknown", size: null }, + fields: { harness: "Codex", model: "unknown", effort: "unknown", size: null, arm: null }, }, ], }); @@ -1875,6 +1875,7 @@ test("an Implementer line reads as agents write it", () => { model: "claude-opus-5-5", effort, size: null, + arm: null, }); expect(read("Claude Code, Opus 5.5, high effort")).toEqual(opus("high")); expect(read("Claude Code, Claude Opus 5.5, xhigh effort")).toEqual(opus("xhigh")); @@ -1885,21 +1886,31 @@ test("an Implementer line reads as agents write it", () => { model: "claude-haiku-5-5", effort: "low", size: null, + arm: null, }); expect(read("claude-haiku-5-5, effort not reported")).toEqual({ harness: "unknown", model: "claude-haiku-5-5", effort: "unknown", size: null, + arm: null, }); expect(read("Codex, GPT-6.1 Sol, medium effort (iterate, XS)")).toEqual({ harness: "Codex", model: "gpt-6.1-sol", effort: "medium", size: "XS", + arm: null, }); expect(read("Claude Code, Opus 5.5, low effort (iterate, flow fix)")).toEqual(opus("low")); - expect(read("Claude Code, Opus 5.5, high effort (iterate, XS, arm B)")).toEqual({ ...opus("high"), size: "XS" }); + expect(read("Claude Code, Opus 5.5, high effort (iterate, XS, arm B)")).toEqual({ + ...opus("high"), + size: "XS", + arm: "B", + }); + expect(read("Claude Code, Opus 5.5, high effort (arm A)")).toEqual({ ...opus("high"), arm: "A" }); + // Only an arm the A/B runs reads as one. + expect(read("Claude Code, Opus 5.5, high effort (arm C)")).toEqual(opus("high")); expect(read("unknown, Opus 5.5, high effort")).toEqual(opus("high", "unknown")); expect(read("Claude Code, Fable 5.2, medium effort")?.model).toBe("claude-fable-5-2"); expect(read("Codex, openai/gpt-6.1-sol, high effort")?.model).toBe("gpt-6.1-sol"); diff --git a/test/usage.test.ts b/test/usage.test.ts index f8f076c2..24ba4bc2 100644 --- a/test/usage.test.ts +++ b/test/usage.test.ts @@ -361,12 +361,14 @@ describe("PR records", () => { model: "claude-opus-5-5", effort: "low", size: "XS", + arm: null, }); expect(implementer("Implementer: Codex, gpt-6.1-sol, unknown effort")).toEqual({ harness: "Codex", model: "gpt-6.1-sol", effort: "unknown", size: null, + arm: null, }); expect(implementer("Implementer: someone")).toBeNull(); }); @@ -394,7 +396,8 @@ describe("usage", () => { ...extra, }); - function setup() { + /** The fixture repository and its PRs and sessions, with `more` PRs and exported sessions besides. */ + function setup(more: { pulls?: object[]; sessions?: object[] } = {}) { const dir = tempDir(); const root = join(dir, "repo"); mkdirSync(root); @@ -452,6 +455,7 @@ describe("usage", () => { cwd: "/both", repositories: ["https://github.com/fork/proj.git", "https://github.com/acme/proj.git"], }), + ...(more.sessions ?? []), ); const body10 = [ @@ -502,7 +506,9 @@ describe("usage", () => { const body11 = "Fix.\n\n## Caused by\n\n- [acme/proj#10](https://github.com/acme/proj/pull/10) - missed a case\n"; // A closed duplicate of #11's fix: it never landed, so it fixed nothing. + // Newest updated first, as GitHub lists them, so the report stops at #8. const pulls = [ + ...(more.pulls ?? []), pr(11, "fix/a", body11, "2026-11-06T20:00:00Z"), pr(10, "quest/m0/a", body10, "2026-11-06T10:00:00Z"), // The same branch's earlier claim, closed before any session reached the branch. @@ -682,9 +688,9 @@ describe("usage", () => { expect(report.folded).toEqual({ copies: 1, hosts: ["box"] }); expect(report.incomplete).toEqual([{ host: null, harness: "claude", session: GONE, skippedLines: 2 }]); expect(ten.implementers).toEqual([ - { harness: "Claude Code", model: "claude-opus-5-5", effort: "high", size: null }, - { harness: "Claude Code", model: "claude-opus-5-5", effort: "high", size: "L" }, - { harness: "Codex", model: "gpt-6.1-sol", effort: "medium", size: "S" }, + { harness: "Claude Code", model: "claude-opus-5-5", effort: "high", size: null, arm: null }, + { harness: "Claude Code", model: "claude-opus-5-5", effort: "high", size: "L", arm: null }, + { harness: "Codex", model: "gpt-6.1-sol", effort: "medium", size: "S", arm: null }, ]); expect(ten.unread).toEqual(["Implementer: Claude Code, Some Model"]); expect(usageLines(report)).toContain(" unread: Implementer: Claude Code, Some Model"); @@ -804,6 +810,91 @@ describe("usage", () => { expect(JSON.stringify(report)).not.toContain("SECRET"); }); + test("rolls up the writer A/B by size and arm, leaving out a PR a writer ran off its arm", async () => { + const models = (...lines: string[]) => + [ + "## Models", + "", + ...lines, + "", + "## Review", + "", + "Rewritten by `quest review` after each round.", + "", + "- Reviewer: Codex, gpt-6.1-sol, high effort", + "- Score: 8/10, `Ok.`", + "- Rounds: 2; findings 1 (medium 1): fixed 1, refuted 0, open 0", + "", + ].join("\n"); + const armed = (number: number, branch: string, ...lines: string[]) => + pr(number, branch, models(...lines), "2026-11-06T00:00:00Z", { updated_at: "2026-11-06T00:00:00Z" }); + const { root, deps, sessions } = setup({ + pulls: [ + armed( + 20, + "quest/m0/b1", + "- Size: M", + "- Implementer: Claude Code, claude-opus-5-5, high effort (arm B)", + "- Implementer: Claude Code, claude-opus-5-5, high effort (iterate, XS, arm B)", + ), + armed( + 21, + "quest/m0/b2", + "- Size: M", + "- Implementer: Claude Code, claude-opus-5-5, high effort (arm B)", + ), + armed( + 22, + "quest/m0/a1", + "- Size: M", + "- Implementer: Claude Code, claude-opus-5-5, medium effort (arm A)", + ), + // A fallback to Sol names no arm, and an iterate naming the other arm, so neither PR counts. + armed(23, "quest/m0/off", "- Size: M", "- Implementer: Codex, gpt-6.1-sol, high effort"), + armed( + 24, + "quest/m0/mixed", + "- Size: M", + "- Implementer: Claude Code, claude-opus-5-5, high effort (arm B)", + "- Implementer: Claude Code, claude-opus-5-5, medium effort (iterate, M, arm A)", + ), + // An unsized PR has no cell to count in. + armed(25, "quest/m0/unsized", "- Implementer: Claude Code, claude-opus-5-5, high effort (arm B)"), + ], + sessions: [ + exported("33333333-3333-3333-3333-333333333333", "claude-opus-5-5", "high", 3, { + branches: [{ name: "quest/m0/b1", first: "2026-11-06T10:05:00Z", last: "2026-11-06T10:06:00Z" }], + cwd: "/b1", + }), + ], + }); + const report = await usage(root, { since: "2026-11-05" }, deps(sessions)); + expect( + report.arms.map((a) => [ + a.size, + a.arm, + a.model, + a.effort, + a.pulls, + a.reviewed, + a.tally.rounds, + a.sessions, + a.spend.cost, + ]), + ).toEqual([ + ["M", "A", "claude-opus-5-5", "medium", 1, 1, 2, 0, 0], + ["M", "B", "claude-opus-5-5", "high", 2, 2, 4, 1, 3], + ]); + const twenty = report.pulls.find((p) => p.number === 20)!; + expect(twenty.implementers.map((i) => i.arm)).toEqual(["B", "B"]); + const text = usageLines(report).join("\n"); + expect(text).toContain(" implementer: Claude Code, claude-opus-5-5, high effort (XS, arm B)\n"); + expect(text).toContain("Writer A/B by size and arm:\n"); + expect(text).toContain( + " M B Claude Code claude-opus-5-5 high 2 PRs, 2 merged, 0 handed back, 2 reviewed: rounds 2.0, findings 2 (medium 2): fixed 2, refuted 0, open 0; 0 later fixes; 1 sessions, in 10, cache 20 written 30 read, out 40, ~$3.00\n", + ); + }); + test("a folded copy still names its directory's repository, and a session's gaps reach each model", async () => { const { root, deps } = setup(); // On `box`, a deleted checkout's only repository evidence is a copy that folds into this host's costlier one.