From 266819197e4a7dd7e9c048f128b7d49a72d605e3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 12:38:20 -0800 Subject: [PATCH 1/4] feat(models)!: S writes on Sonnet high, XL on Fable medium, and nothing runs max Co-Authored-By: Claude Opus 5.5 --- README.md | 16 ++++++++++------ quest/m0/routing-evidence/writer-ab.md | 9 +++++++-- quest/m1/model-check.md | 9 +++++++++ src/cli/main.ts | 21 +++++++++++++++++---- src/cli/models.ts | 9 ++++++--- src/cli/review.ts | 4 ++-- test/agent.test.ts | 7 +++++++ test/cli.test.ts | 11 ++++++----- test/review.test.ts | 8 +++++++- 9 files changed, 71 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index 635050a9..dbbe9a42 100644 --- a/README.md +++ b/README.md @@ -133,7 +133,7 @@ $ quest --root examples/export unblocked 1 quest/m0/export.md $ quest --root examples/export unblocked --json --limit 1 -[{"path":"quest/m0/export.md","dependents":1,"hosts":[],"models":[{"harness":"claude","model":"claude-opus-5-5","effort":"low"},{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}]}] +[{"path":"quest/m0/export.md","dependents":1,"hosts":[],"models":[{"harness":"claude","model":"claude-sonnet-5-5","effort":"high"},{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}]}] ``` Each plain output line contains the count, a tab, and the path. On a terminal, @@ -1341,10 +1341,10 @@ the same for every repository unless its `AGENTS.md` names others: | Size | Writer | Fallback | |---|---|---| | XS | Claude Code `claude-sonnet-5-5` high | Codex `gpt-6.1-sol` low | -| S | Claude Code `claude-opus-5-5` low | Codex `gpt-6.1-sol` medium | +| S | Claude Code `claude-sonnet-5-5` high | Codex `gpt-6.1-sol` medium | | M | Claude Code `claude-opus-5-5` medium | Codex `gpt-6.1-sol` high | | L | Claude Code `claude-opus-5-5` high | Codex `gpt-6.1-sol` xhigh | -| XL | Claude Code `claude-opus-5-5` xhigh | none | +| XL | Claude Code `claude-fable-5-2` medium | Claude Code `claude-opus-5-5` xhigh | `quest unblocked --json` lists each ready quest's ranked `models`: its size's list, or the next size's while a `Maintainer:` item under its `## Waiting` @@ -1366,14 +1366,14 @@ claude claude-sonnet-5-5 high codex gpt-6.1-sol low $ quest models XS --writer claude-opus-5-5 -claude claude-opus-5-5 low +claude claude-opus-5-5 medium ``` `--writer` prints the size's entry for that model, or else its entry at the nearest size that lists it, the larger on a tie; a model no list names is an error. `--json` prints an array of `{harness, model, effort}` objects. The iterate records the size it was given in its `Implementer:` line, as -`Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS)`, so +`Implementer: Claude Code, claude-opus-5-5, medium effort (iterate, XS)`, so the model check can audit the routing. `quest review` still reviews the whole branch at the PR's size. @@ -1397,7 +1397,7 @@ family no writer runs, at about $0.05 a review. The shadow, the pick's model one effort lower (Sol low for an XS PR), reviews it too, in its own session, so the routing can be retuned from what each effort step finds over the one below it. It is skipped when the pick runs at low or at an `--effort` off the ladder -(low, medium, high, xhigh, max). It runs on the branch's first run of the +(low, medium, high, xhigh). It runs on the branch's first run of the pair, then reruns only while it has findings to settle, or once the pick changed so that it is no longer one step below; its findings are numbered, answered, and settled like any reviewer's. A pick with no step below it seats @@ -1701,6 +1701,10 @@ quest agent stop --all quest agent admit ``` +`--effort` takes any level the CLI does except `max`, which costs about as much +as a stronger model at a lower effort without matching it; `quest review` +refuses it too. + Codex starts and resumes only in a checkout registered by `quest worktree create`, on Linux with the `quest worktree run` supervisor available. Ownership and host support are checked before any agent state is written. Each turn uses diff --git a/quest/m0/routing-evidence/writer-ab.md b/quest/m0/routing-evidence/writer-ab.md index 11ab9623..71dadfa6 100644 --- a/quest/m0/routing-evidence/writer-ab.md +++ b/quest/m0/routing-evidence/writer-ab.md @@ -3,8 +3,8 @@ ## Goal `quest unblocked --json` routes each XS quest to Sonnet 5.5 high or Opus -low, each S quest to Opus low or medium, and each M quest to Opus medium -or high, 50/50. The split is stable per quest, +low, each S quest to Sonnet 5.5 high or Opus medium, and each M quest to +Opus medium or high, 50/50. The split is stable per quest, and the review efforts in `src/cli/models.ts` stay fixed while it runs. Each `Implementer:` line records its arm, iterates stay on the PR's arm, and `quest usage` reports cost, rounds, findings, hand-backs, and later @@ -37,6 +37,11 @@ Decided 2026-11-11 (user, /quest-plan, - Rejected: Sonnet medium against Sonnet high, since the same cost guide puts Sonnet medium about 7 points below high on its SWE-bench Pro subset and 15 below on its internal agentic-coding set; no XS arm +- Re-armed 2026-11-13 (user, after reading Anthropic's + [model guide](https://claude.dev/blog/which-claude-model-should-you-build-with/)): + S writes on Sonnet 5.5 high instead of Opus low, so its arms are Sonnet + high against Opus medium, the guide's default. Rejected: dropping the S + arm; keeping S on Opus low until this reports. - Escapes are too rare to separate the arms, about 2% of PRs. Compare first-round findings under the fixed review efforts, rounds, hand-backs, and cost, and report escapes alongside. diff --git a/quest/m1/model-check.md b/quest/m1/model-check.md index c264481c..cc9149a3 100644 --- a/quest/m1/model-check.md +++ b/quest/m1/model-check.md @@ -111,6 +111,15 @@ checkpoints and outages to count by hand: puts Sonnet 5.5 high level with Opus 5.5 low (87.6% against 87.4%) at about the same cost per solved task, and medium about 7 points below high on its SWE-bench Pro subset. + - Changed again (maintainer, 2026-11-13), on Anthropic's + [model guide](https://claude.dev/blog/which-claude-model-should-you-build-with/), + which puts scoped coding on Sonnet 5.5 and advises moving up a model + rather than past `high`: S writes on Sonnet 5.5 high instead of Opus + low, whose S PRs averaged 4.7 review rounds over 6 reviewed; XL writes + on Fable 5.2 medium, with Opus xhigh as its fallback, reversing the + 2026-11-09 rejection; and nothing runs `max`, which `quest agent` and + `quest review` refuse. XS stays Sonnet high, since `quest usage` had no + Sonnet medium runs to compare. Tally the S and XL cells fresh. - Compare both tables' models (in `src/cli/models.ts`) with each vendor's current models: the newest model of each tier the tables pin. Bump any that are stale, and check that both CLIs accept the new ids and the pinned efforts. `--deep` and its strongest-model diff --git a/src/cli/main.ts b/src/cli/main.ts index 8a681772..0cc8dd69 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -1047,7 +1047,7 @@ export function main(argv: string[], out: Output): number | Promise { command .addOption(new Option("--harness ", "Agent CLI to run").choices(HARNESSES).makeOptionMandatory()) .requiredOption("--model ", "Model the agent runs") - .requiredOption("--effort ", "Reasoning effort the agent runs at") + .requiredOption("--effort ", "Reasoning effort the agent runs at, never max", effortLevel) .requiredOption( "--permission-mode ", "Claude Code's --permission-mode, or Codex's sandbox mode; the agent cannot prompt", @@ -1132,7 +1132,7 @@ export function main(argv: string[], out: Output): number | Promise { ". A spawn starts a quest on the first entry that can run, as quest unblocked --json lists them. " + "--writer prints the one entry that runs that model on the work instead, for an iterate, which stays " + "on its PR's writer: the size's own entry for the model, else its entry at the nearest size that " + - "lists it, the larger on a tie, so an XS iterate on an Opus PR runs Opus at S's effort. A model no " + + "lists it, the larger on a tie, so an XS iterate on an Opus PR runs Opus at M's effort. A model no " + "list names is an error. Prints harness, model, and effort per line, tab-separated; --json prints an " + "array of {harness, model, effort} objects.", ) @@ -1232,7 +1232,7 @@ export function main(argv: string[], out: Output): number | Promise { `model that changed it last. Each size ranks its reviewers (harness, model, effort): ${reviewers}. The ` + "first whose model is not avoided reviews, and beside it, on the same head and each in its own session, " + `${SECOND_REVIEWER.harness} ${SECOND_REVIEWER.model} ${SECOND_REVIEWER.effort} for every size, and the shadow: ` + - "the pick's model one effort lower (low, medium, high, xhigh, max), skipped when the pick runs at low or an " + + "the pick's model one effort lower (low, medium, high, xhigh), skipped when the pick runs at low or an " + "effort off that ladder, so each effort step can be compared with the one below it. The shadow runs on a " + "branch's first run of the pair, then only while it has findings to settle or once the pick changed so it is " + "no longer one step below; its findings are numbered and settled like any reviewer's, its session carries " + @@ -1325,7 +1325,11 @@ export function main(argv: string[], out: Output): number | Promise { ).choices(REVIEW_HARNESSES), ) .option("--model ", "Model the size's pick, or --harness's reviewer, runs instead") - .option("--effort ", "Reasoning effort the size's pick, or --harness's reviewer, runs at instead") + .option( + "--effort ", + "Reasoning effort the size's pick, or --harness's reviewer, runs at instead, never max", + effortLevel, + ) .option( "--exclude-model ", "A model never picked to review it, instead of the PR's latest writer; repeat for each", @@ -1645,6 +1649,15 @@ export function main(argv: string[], out: Output): number | Promise { } } +/** + * An agent's effort: any level but `max`, which costs as much as a stronger + * model at a lower effort without matching it. + */ +function effortLevel(value: string): string { + if (value === "max") throw new InvalidArgumentError("max is never run; use a stronger model instead"); + return nonEmpty(value); +} + /** An empty path names no directory, so it is a usage error rather than `.`. */ function nonEmpty(value: string): string { if (value === "") throw new InvalidArgumentError("a value is required"); diff --git a/src/cli/models.ts b/src/cli/models.ts index 0bed0272..9a8eec31 100644 --- a/src/cli/models.ts +++ b/src/cli/models.ts @@ -19,6 +19,7 @@ export const DEFAULT_SIZE: Size = "M"; const sol = (effort: string): Model => ({ harness: "codex", model: "gpt-6.1-sol", effort }); const opus = (effort: string): Model => ({ harness: "claude", model: "claude-opus-5-5", effort }); const sonnet = (effort: string): Model => ({ harness: "claude", model: "claude-sonnet-5-5", effort }); +const fable = (effort: string): Model => ({ harness: "claude", model: "claude-fable-5-2", effort }); /** * Each size's writers, ranked: a spawn starts the quest on the first that can @@ -26,14 +27,16 @@ const sonnet = (effort: string): Model => ({ harness: "claude", model: "claude-s * writes a PR runs all its iterates. XS runs Sonnet rather than the cheaper * Haiku, whose surcharge past 100k input tokens leads users to cap its * compaction window there, barely above an agent's fixed load of prompt, - * context, and skill. + * context, and skill. S runs scoped work on Sonnet, at half Opus's price, and + * XL moves up to Fable rather than push Opus past high, as Anthropic's model + * guide advises. */ export const WRITERS: Record = { XS: [sonnet("high"), sol("low")], - S: [opus("low"), sol("medium")], + S: [sonnet("high"), sol("medium")], M: [opus("medium"), sol("high")], L: [opus("high"), sol("xhigh")], - XL: [opus("xhigh")], + XL: [fable("medium"), opus("xhigh")], }; /** diff --git a/src/cli/review.ts b/src/cli/review.ts index e419f369..5647f659 100644 --- a/src/cli/review.ts +++ b/src/cli/review.ts @@ -1430,8 +1430,8 @@ const seatKey = (seat: SeatId) => `${seat.reviewer}${seat.shadow ? " shadow" : " /** Whether two reviewers hold the same seat: the same harness, both the shadow or neither. */ export const sameSeat = (a: SeatId, b: SeatId) => seatKey(a) === seatKey(b); -/** The efforts the harnesses take, from lowest to highest. */ -const EFFORTS = ["low", "medium", "high", "xhigh", "max"]; +/** The efforts a reviewer runs at, from lowest to highest; `quest` never runs `max`. */ +const EFFORTS = ["low", "medium", "high", "xhigh"]; /** * The shadow of `routed`, the size's pick: its model one effort lower, or null when no lower effort is known, diff --git a/test/agent.test.ts b/test/agent.test.ts index 30e26637..3fa5ceb0 100644 --- a/test/agent.test.ts +++ b/test/agent.test.ts @@ -300,6 +300,13 @@ test("on Windows, the agent environment drops credentials whatever their case", expect(strippedEnvironment({ op_token: "t" }, "linux")).toEqual({ op_token: "t" }); }); +test("start refuses max effort before launching anything", async () => { + const args = launch("claude").map((arg) => (arg === "high" ? "max" : arg)); + const result = await quest("agent", "start", ...args, ".", "go"); + expect(result.code).toBe(2); + expect(result.stderr).toContain("max is never run"); +}); + test("start runs claude with every launch setting on its own login", async () => { const s = setup(); const result = await quest("agent", "start", ...launch("claude"), s.worktree, "/quest-start quest/m0/a.md"); diff --git a/test/cli.test.ts b/test/cli.test.ts index 1be00ef2..e58e7ad2 100644 --- a/test/cli.test.ts +++ b/test/cli.test.ts @@ -152,11 +152,11 @@ test("a document that is not UTF-8 is an error", () => { test("unblocked prints ranked counts and paths, with deterministic JSON and a limit", () => { const tree = Tree.baseline(); expect(tree.run("unblocked")).toEqual({ code: 0, stdout: "1\tquest/m0/epic/one.md\n", stderr: "" }); - const opusLow = '{"harness":"claude","model":"claude-opus-5-5","effort":"low"}'; + const sonnetHigh = '{"harness":"claude","model":"claude-sonnet-5-5","effort":"high"}'; const solMedium = '{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}'; expect(tree.run("unblocked", "--json", "--limit", "1")).toEqual({ code: 0, - stdout: `[{"path":"quest/m0/epic/one.md","dependents":1,"hosts":[],"models":[${opusLow},${solMedium}]}]\n`, + stdout: `[{"path":"quest/m0/epic/one.md","dependents":1,"hosts":[],"models":[${sonnetHigh},${solMedium}]}]\n`, stderr: "", }); tree.write("quest/m0/epic/two.md", "# [S] Two\n\n## Goal\n\nWork.\n"); @@ -175,11 +175,11 @@ test("unblocked routes a quest with a decision open one size up, and XL stays XL quest.models.map((entry) => entry.effort).join(" "), ); const tree = Tree.baseline().append("quest/m0/epic/one.md", "\n## Waiting\n\n- moq-dev/moq: a release.\n"); - expect(models(tree)).toEqual(["low medium"]); + expect(models(tree)).toEqual(["high medium"]); tree.append("quest/m0/epic/one.md", "- Maintainer: pick a name.\n"); expect(models(tree)).toEqual(["medium high"]); tree.write("quest/m0/epic/one.md", "# [XL] One\n\n## Goal\n\nWork.\n\n## Waiting\n\n- Maintainer: pick.\n"); - expect(models(tree)).toEqual(["xhigh"]); + expect(models(tree)).toEqual(["medium xhigh"]); }); test("models ranks a size's writers, or the one entry that runs a PR's writer", () => { @@ -190,12 +190,13 @@ test("models ranks a size's writers, or the one entry that runs a PR's writer", stderr: "", }); expect(JSON.parse(tree.run("models", "XL", "--json").stdout)).toEqual([ + { harness: "claude", model: "claude-fable-5-2", effort: "medium" }, { harness: "claude", model: "claude-opus-5-5", effort: "xhigh" }, ]); const writer = (size: string, model: string) => tree.run("models", size, "--writer", model).stdout; expect(writer("M", "gpt-6.1-sol")).toBe("codex\tgpt-6.1-sol\thigh\n"); // A writer its size does not list runs at the nearest size that lists it. - expect(writer("XS", "claude-opus-5-5")).toBe("claude\tclaude-opus-5-5\tlow\n"); + expect(writer("XS", "claude-opus-5-5")).toBe("claude\tclaude-opus-5-5\tmedium\n"); expect(writer("L", "claude-sonnet-5-5")).toBe("claude\tclaude-sonnet-5-5\thigh\n"); expect(writer("XL", "gpt-6.1-sol")).toBe("codex\tgpt-6.1-sol\txhigh\n"); const unknown = tree.run("models", "S", "--writer", "unknown"); diff --git a/test/review.test.ts b/test/review.test.ts index b4ff46ab..88d74bde 100644 --- a/test/review.test.ts +++ b/test/review.test.ts @@ -1136,6 +1136,12 @@ fakes("--base reviews against another branch, fetched first, with that branch's expect(unknown.stderr).toContain("--base must be /"); }); +test("review refuses max effort", async () => { + const result = await quest("review", "--harness", "claude", "--model", "opus", "--effort", "max"); + expect(result.code).toBe(2); + expect(result.stderr).toContain("max is never run"); +}); + fakes("claude reviews read-only with the base's instructions, then reports in the schema", async () => { const s = setup("CODEX_THREAD_ID"); // The rules travel inside the command's one double-quoted argument. @@ -1998,7 +2004,7 @@ test("the shadow is the size's pick one effort lower, and none below low", () => "claude claude-opus-5-5 medium", "claude claude-opus-5-5 high", ]); - expect(shadowOf({ harness: "claude", model: "m", effort: "max" })!.effort).toBe("xhigh"); + expect(shadowOf({ harness: "claude", model: "m", effort: "xhigh" })!.effort).toBe("high"); // No step below an effort off the ladder is known, so an --effort such as Codex's minimal runs no shadow. expect(shadowOf({ harness: "codex", model: "m", effort: "minimal" })).toBeNull(); }); From 806a0f6390da97db9b5bab1bed76e5fa8dddf68a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 12:42:45 -0800 Subject: [PATCH 2/4] feat(models): S writes on Opus medium; A/B the disputed writer picks Co-Authored-By: Claude Opus 5.5 --- README.md | 4 +-- quest/m0/routing-evidence/fable-unstick.md | 37 ++++++++++++++++++++++ quest/m0/routing-evidence/writer-ab.md | 32 +++++++++++++------ quest/m1/model-check.md | 10 +++--- src/cli/main.ts | 2 +- src/cli/models.ts | 6 ++-- test/cli.test.ts | 6 ++-- 7 files changed, 75 insertions(+), 22 deletions(-) create mode 100644 quest/m0/routing-evidence/fable-unstick.md diff --git a/README.md b/README.md index dbbe9a42..5827c11a 100644 --- a/README.md +++ b/README.md @@ -133,7 +133,7 @@ $ quest --root examples/export unblocked 1 quest/m0/export.md $ quest --root examples/export unblocked --json --limit 1 -[{"path":"quest/m0/export.md","dependents":1,"hosts":[],"models":[{"harness":"claude","model":"claude-sonnet-5-5","effort":"high"},{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}]}] +[{"path":"quest/m0/export.md","dependents":1,"hosts":[],"models":[{"harness":"claude","model":"claude-opus-5-5","effort":"medium"},{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}]}] ``` Each plain output line contains the count, a tab, and the path. On a terminal, @@ -1341,7 +1341,7 @@ the same for every repository unless its `AGENTS.md` names others: | Size | Writer | Fallback | |---|---|---| | XS | Claude Code `claude-sonnet-5-5` high | Codex `gpt-6.1-sol` low | -| S | Claude Code `claude-sonnet-5-5` high | Codex `gpt-6.1-sol` medium | +| S | Claude Code `claude-opus-5-5` medium | Codex `gpt-6.1-sol` medium | | M | Claude Code `claude-opus-5-5` medium | Codex `gpt-6.1-sol` high | | L | Claude Code `claude-opus-5-5` high | Codex `gpt-6.1-sol` xhigh | | XL | Claude Code `claude-fable-5-2` medium | Claude Code `claude-opus-5-5` xhigh | diff --git a/quest/m0/routing-evidence/fable-unstick.md b/quest/m0/routing-evidence/fable-unstick.md new file mode 100644 index 00000000..cb3d1caf --- /dev/null +++ b/quest/m0/routing-evidence/fable-unstick.md @@ -0,0 +1,37 @@ +# [M] Hand looping PRs to Fable in half the cases + +## Goal + +A `/quest-iterate` on a PR whose review has run past a threshold of rounds +runs, 50/50 and stable per PR, either on the PR's writer as today or on +Fable 5.2 medium. Its `Implementer:` line records the arm, and `quest usage` +reports the rounds left to merge, findings fixed and refuted, hand-backs, +later culprits, and cost per arm, for the +[routing evidence](/quest/m0/routing-evidence/README.md) readout. + +## Plan + +Follow-up of the 2026-11-13 routing change (user). Theo Browne's +[Opus against Fable video](https://www.youtube.com/watch?v=XtykeVoSmP8) +finds that Opus applies every review finding, so a small PR keeps growing, +while Fable fixes what Opus could not or judges the rest good enough to +merge. Opus low on S quests averaged 4.7 review rounds over 6 reviewed PRs +in `quest usage` since 2026-10-01. + +Agent's choices, for the implementer to revisit: + +- Threshold: past round 4. Most PRs merge by then, so the arms take only + the loops. +- Read the round count from the `## Review` record's tally, as `quest + usage` already does, and pick the arm from a hash of the PR's branch, as + [the writer A/B](/quest/m0/routing-evidence/writer-ab.md) does for + quests. +- Fable is a Claude model, so the reviewer pick still avoids it when it + wrote last; Sol keeps reviewing such a PR. +- Record the arm on the Implementer line as `(iterate, S, unstick)` or + `(iterate, S, writer)`. + +## Required + +- [usage through token-max](/quest/m0/routing-evidence/usage-token-max.md) - + counts sub-agent writers' cost, without which an arm's cost is undercounted diff --git a/quest/m0/routing-evidence/writer-ab.md b/quest/m0/routing-evidence/writer-ab.md index 71dadfa6..823765fc 100644 --- a/quest/m0/routing-evidence/writer-ab.md +++ b/quest/m0/routing-evidence/writer-ab.md @@ -1,10 +1,10 @@ -# [M] Split XS, S and M writers between two arms +# [M] Split each size's writers between two arms ## Goal -`quest unblocked --json` routes each XS quest to Sonnet 5.5 high or Opus -low, each S quest to Sonnet 5.5 high or Opus medium, and each M quest to -Opus medium or high, 50/50. The split is stable per quest, +`quest unblocked --json` routes each XS and S quest to Sonnet 5.5 high or +Opus 5.5 medium, each M quest to Opus medium or high, and each XL quest to +Fable 5.2 medium or Opus xhigh, 50/50. The split is stable per quest, and the review efforts in `src/cli/models.ts` stay fixed while it runs. Each `Implementer:` line records its arm, iterates stay on the PR's arm, and `quest usage` reports cost, rounds, findings, hand-backs, and later @@ -37,11 +37,25 @@ Decided 2026-11-11 (user, /quest-plan, - Rejected: Sonnet medium against Sonnet high, since the same cost guide puts Sonnet medium about 7 points below high on its SWE-bench Pro subset and 15 below on its internal agentic-coding set; no XS arm -- Re-armed 2026-11-13 (user, after reading Anthropic's - [model guide](https://claude.dev/blog/which-claude-model-should-you-build-with/)): - S writes on Sonnet 5.5 high instead of Opus low, so its arms are Sonnet - high against Opus medium, the guide's default. Rejected: dropping the S - arm; keeping S on Opus low until this reports. +- Re-armed 2026-11-13 (user), to test where Anthropic's + [model guide](https://claude.dev/blog/which-claude-model-should-you-build-with/) + and Theo Browne's + [Opus against Fable video](https://www.youtube.com/watch?v=XtykeVoSmP8) + disagree: + - S Sonnet high against Opus medium: the guide puts scoped coding on + Sonnet, Theo finds Sonnet runs up output tokens and costs about what + Opus does. S's table entry moved from Opus low to Opus medium, the + guide's default, since both call Opus low too weak. + - XS Sonnet high against Opus medium rather than Opus low, the same + contrast as S (agent's choice, since no table entry runs Opus low any + more). + - M Opus medium against high, as before: Theo finds medium ends its turn + too early and calls high the best effort. + - XL Fable medium against Opus xhigh: the guide moves up to Fable rather + than past high, while Theo finds Opus checks verifiable work more + carefully and a subscription gives about a quarter as much Fable. XL + quests are rare, so report how many each arm got rather than wait for + a detectable difference. - Escapes are too rare to separate the arms, about 2% of PRs. Compare first-round findings under the fixed review efforts, rounds, hand-backs, and cost, and report escapes alongside. diff --git a/quest/m1/model-check.md b/quest/m1/model-check.md index cc9149a3..7438b367 100644 --- a/quest/m1/model-check.md +++ b/quest/m1/model-check.md @@ -114,12 +114,14 @@ checkpoints and outages to count by hand: - Changed again (maintainer, 2026-11-13), on Anthropic's [model guide](https://claude.dev/blog/which-claude-model-should-you-build-with/), which puts scoped coding on Sonnet 5.5 and advises moving up a model - rather than past `high`: S writes on Sonnet 5.5 high instead of Opus - low, whose S PRs averaged 4.7 review rounds over 6 reviewed; XL writes - on Fable 5.2 medium, with Opus xhigh as its fallback, reversing the + rather than past `high`: S writes on Opus 5.5 medium instead of low, + whose S PRs averaged 4.7 review rounds over 6 reviewed; XL writes on + Fable 5.2 medium, with Opus xhigh as its fallback, reversing the 2026-11-09 rejection; and nothing runs `max`, which `quest agent` and `quest review` refuse. XS stays Sonnet high, since `quest usage` had no - Sonnet medium runs to compare. Tally the S and XL cells fresh. + Sonnet medium runs to compare. The + [writer A/B](/quest/m0/routing-evidence/writer-ab.md) tests each change + against its alternative; tally the S and XL cells fresh. - Compare both tables' models (in `src/cli/models.ts`) with each vendor's current models: the newest model of each tier the tables pin. Bump any that are stale, and check that both CLIs accept the new ids and the pinned efforts. `--deep` and its strongest-model diff --git a/src/cli/main.ts b/src/cli/main.ts index 0cc8dd69..f5c4633e 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -1132,7 +1132,7 @@ export function main(argv: string[], out: Output): number | Promise { ". A spawn starts a quest on the first entry that can run, as quest unblocked --json lists them. " + "--writer prints the one entry that runs that model on the work instead, for an iterate, which stays " + "on its PR's writer: the size's own entry for the model, else its entry at the nearest size that " + - "lists it, the larger on a tie, so an XS iterate on an Opus PR runs Opus at M's effort. A model no " + + "lists it, the larger on a tie, so an XS iterate on an Opus PR runs Opus at S's effort. A model no " + "list names is an error. Prints harness, model, and effort per line, tab-separated; --json prints an " + "array of {harness, model, effort} objects.", ) diff --git a/src/cli/models.ts b/src/cli/models.ts index 9a8eec31..fc1f5717 100644 --- a/src/cli/models.ts +++ b/src/cli/models.ts @@ -27,13 +27,13 @@ const fable = (effort: string): Model => ({ harness: "claude", model: "claude-fa * writes a PR runs all its iterates. XS runs Sonnet rather than the cheaper * Haiku, whose surcharge past 100k input tokens leads users to cap its * compaction window there, barely above an agent's fixed load of prompt, - * context, and skill. S runs scoped work on Sonnet, at half Opus's price, and - * XL moves up to Fable rather than push Opus past high, as Anthropic's model + * context, and skill. No size runs Opus at low, which misses too much, and XL + * moves up to Fable rather than push Opus past high, as Anthropic's model * guide advises. */ export const WRITERS: Record = { XS: [sonnet("high"), sol("low")], - S: [sonnet("high"), sol("medium")], + S: [opus("medium"), sol("medium")], M: [opus("medium"), sol("high")], L: [opus("high"), sol("xhigh")], XL: [fable("medium"), opus("xhigh")], diff --git a/test/cli.test.ts b/test/cli.test.ts index e58e7ad2..df843651 100644 --- a/test/cli.test.ts +++ b/test/cli.test.ts @@ -152,11 +152,11 @@ test("a document that is not UTF-8 is an error", () => { test("unblocked prints ranked counts and paths, with deterministic JSON and a limit", () => { const tree = Tree.baseline(); expect(tree.run("unblocked")).toEqual({ code: 0, stdout: "1\tquest/m0/epic/one.md\n", stderr: "" }); - const sonnetHigh = '{"harness":"claude","model":"claude-sonnet-5-5","effort":"high"}'; + const opusMedium = '{"harness":"claude","model":"claude-opus-5-5","effort":"medium"}'; const solMedium = '{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}'; expect(tree.run("unblocked", "--json", "--limit", "1")).toEqual({ code: 0, - stdout: `[{"path":"quest/m0/epic/one.md","dependents":1,"hosts":[],"models":[${sonnetHigh},${solMedium}]}]\n`, + stdout: `[{"path":"quest/m0/epic/one.md","dependents":1,"hosts":[],"models":[${opusMedium},${solMedium}]}]\n`, stderr: "", }); tree.write("quest/m0/epic/two.md", "# [S] Two\n\n## Goal\n\nWork.\n"); @@ -175,7 +175,7 @@ test("unblocked routes a quest with a decision open one size up, and XL stays XL quest.models.map((entry) => entry.effort).join(" "), ); const tree = Tree.baseline().append("quest/m0/epic/one.md", "\n## Waiting\n\n- moq-dev/moq: a release.\n"); - expect(models(tree)).toEqual(["high medium"]); + expect(models(tree)).toEqual(["medium medium"]); tree.append("quest/m0/epic/one.md", "- Maintainer: pick a name.\n"); expect(models(tree)).toEqual(["medium high"]); tree.write("quest/m0/epic/one.md", "# [XL] One\n\n## Goal\n\nWork.\n\n## Waiting\n\n- Maintainer: pick.\n"); From dd6007dd73ef4c714ddc464bc17c71ecf35fb299 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 12:54:59 -0800 Subject: [PATCH 3/4] feat(models): XL stays on Opus xhigh, with Fable as its A/B arm Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- quest/m0/routing-evidence/writer-ab.md | 11 ++++++----- quest/m1/model-check.md | 14 +++++++++----- src/cli/models.ts | 9 ++++----- test/cli.test.ts | 3 +-- 5 files changed, 21 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 5827c11a..df277114 100644 --- a/README.md +++ b/README.md @@ -1344,7 +1344,7 @@ the same for every repository unless its `AGENTS.md` names others: | S | Claude Code `claude-opus-5-5` medium | Codex `gpt-6.1-sol` medium | | M | Claude Code `claude-opus-5-5` medium | Codex `gpt-6.1-sol` high | | L | Claude Code `claude-opus-5-5` high | Codex `gpt-6.1-sol` xhigh | -| XL | Claude Code `claude-fable-5-2` medium | Claude Code `claude-opus-5-5` xhigh | +| XL | Claude Code `claude-opus-5-5` xhigh | none | `quest unblocked --json` lists each ready quest's ranked `models`: its size's list, or the next size's while a `Maintainer:` item under its `## Waiting` diff --git a/quest/m0/routing-evidence/writer-ab.md b/quest/m0/routing-evidence/writer-ab.md index 823765fc..8be35ef2 100644 --- a/quest/m0/routing-evidence/writer-ab.md +++ b/quest/m0/routing-evidence/writer-ab.md @@ -4,7 +4,7 @@ `quest unblocked --json` routes each XS and S quest to Sonnet 5.5 high or Opus 5.5 medium, each M quest to Opus medium or high, and each XL quest to -Fable 5.2 medium or Opus xhigh, 50/50. The split is stable per quest, +Opus xhigh or Fable 5.2 medium, 50/50. The split is stable per quest, and the review efforts in `src/cli/models.ts` stay fixed while it runs. Each `Implementer:` line records its arm, iterates stay on the PR's arm, and `quest usage` reports cost, rounds, findings, hand-backs, and later @@ -51,11 +51,12 @@ Decided 2026-11-11 (user, /quest-plan, more). - M Opus medium against high, as before: Theo finds medium ends its turn too early and calls high the best effort. - - XL Fable medium against Opus xhigh: the guide moves up to Fable rather + - XL Opus xhigh against Fable medium: the guide moves up to Fable rather than past high, while Theo finds Opus checks verifiable work more - carefully and a subscription gives about a quarter as much Fable. XL - quests are rare, so report how many each arm got rather than wait for - a detectable difference. + carefully and a subscription gives about a quarter as much Fable. XL's + table entry stays Opus xhigh for that subscription cap (rejected: + Fable medium; dropping the Fable arm). XL quests are rare, so report + how many each arm got rather than wait for a detectable difference. - Escapes are too rare to separate the arms, about 2% of PRs. Compare first-round findings under the fixed review efforts, rounds, hand-backs, and cost, and report escapes alongside. diff --git a/quest/m1/model-check.md b/quest/m1/model-check.md index 7438b367..5d4f5f61 100644 --- a/quest/m1/model-check.md +++ b/quest/m1/model-check.md @@ -115,13 +115,17 @@ checkpoints and outages to count by hand: [model guide](https://claude.dev/blog/which-claude-model-should-you-build-with/), which puts scoped coding on Sonnet 5.5 and advises moving up a model rather than past `high`: S writes on Opus 5.5 medium instead of low, - whose S PRs averaged 4.7 review rounds over 6 reviewed; XL writes on - Fable 5.2 medium, with Opus xhigh as its fallback, reversing the - 2026-11-09 rejection; and nothing runs `max`, which `quest agent` and - `quest review` refuse. XS stays Sonnet high, since `quest usage` had no + whose S PRs averaged 4.7 review rounds over 6 reviewed; and nothing + runs `max`, which `quest agent` and `quest review` refuse. XL stays on + Opus xhigh: the maintainer first moved it to Fable 5.2 medium, then + kept Opus, since a Max or Ultra plan caps Fable at half its weekly + limit and Fable uses that limit faster (Theo Browne's + [video](https://www.youtube.com/watch?v=XtykeVoSmP8) estimates about + $3k a month of Fable against $14k of Opus on the $200 plan). XS stays Sonnet high, since `quest usage` had no Sonnet medium runs to compare. The [writer A/B](/quest/m0/routing-evidence/writer-ab.md) tests each change - against its alternative; tally the S and XL cells fresh. + against its alternative, Fable medium on XL included; tally the S cell + fresh. - Compare both tables' models (in `src/cli/models.ts`) with each vendor's current models: the newest model of each tier the tables pin. Bump any that are stale, and check that both CLIs accept the new ids and the pinned efforts. `--deep` and its strongest-model diff --git a/src/cli/models.ts b/src/cli/models.ts index fc1f5717..e42024b8 100644 --- a/src/cli/models.ts +++ b/src/cli/models.ts @@ -19,7 +19,6 @@ export const DEFAULT_SIZE: Size = "M"; const sol = (effort: string): Model => ({ harness: "codex", model: "gpt-6.1-sol", effort }); const opus = (effort: string): Model => ({ harness: "claude", model: "claude-opus-5-5", effort }); const sonnet = (effort: string): Model => ({ harness: "claude", model: "claude-sonnet-5-5", effort }); -const fable = (effort: string): Model => ({ harness: "claude", model: "claude-fable-5-2", effort }); /** * Each size's writers, ranked: a spawn starts the quest on the first that can @@ -27,16 +26,16 @@ const fable = (effort: string): Model => ({ harness: "claude", model: "claude-fa * writes a PR runs all its iterates. XS runs Sonnet rather than the cheaper * Haiku, whose surcharge past 100k input tokens leads users to cap its * compaction window there, barely above an agent's fixed load of prompt, - * context, and skill. No size runs Opus at low, which misses too much, and XL - * moves up to Fable rather than push Opus past high, as Anthropic's model - * guide advises. + * context, and skill. No size runs Opus at low, which misses too much. XL + * stays on Opus rather than Fable, which a subscription caps at half its + * weekly limit and which uses that limit faster. */ export const WRITERS: Record = { XS: [sonnet("high"), sol("low")], S: [opus("medium"), sol("medium")], M: [opus("medium"), sol("high")], L: [opus("high"), sol("xhigh")], - XL: [fable("medium"), opus("xhigh")], + XL: [opus("xhigh")], }; /** diff --git a/test/cli.test.ts b/test/cli.test.ts index df843651..39de53a2 100644 --- a/test/cli.test.ts +++ b/test/cli.test.ts @@ -179,7 +179,7 @@ test("unblocked routes a quest with a decision open one size up, and XL stays XL tree.append("quest/m0/epic/one.md", "- Maintainer: pick a name.\n"); expect(models(tree)).toEqual(["medium high"]); tree.write("quest/m0/epic/one.md", "# [XL] One\n\n## Goal\n\nWork.\n\n## Waiting\n\n- Maintainer: pick.\n"); - expect(models(tree)).toEqual(["medium xhigh"]); + expect(models(tree)).toEqual(["xhigh"]); }); test("models ranks a size's writers, or the one entry that runs a PR's writer", () => { @@ -190,7 +190,6 @@ test("models ranks a size's writers, or the one entry that runs a PR's writer", stderr: "", }); expect(JSON.parse(tree.run("models", "XL", "--json").stdout)).toEqual([ - { harness: "claude", model: "claude-fable-5-2", effort: "medium" }, { harness: "claude", model: "claude-opus-5-5", effort: "xhigh" }, ]); const writer = (size: string, model: string) => tree.run("models", size, "--writer", model).stdout; From fe9840070da78e327ba64c7d67cb1894d8cf797e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 13:44:27 -0800 Subject: [PATCH 4/4] Drop the fable-unstick follow-up quest Co-Authored-By: Claude Opus 5.5 --- quest/m0/routing-evidence/fable-unstick.md | 38 ---------------------- 1 file changed, 38 deletions(-) delete mode 100644 quest/m0/routing-evidence/fable-unstick.md diff --git a/quest/m0/routing-evidence/fable-unstick.md b/quest/m0/routing-evidence/fable-unstick.md deleted file mode 100644 index 34defac3..00000000 --- a/quest/m0/routing-evidence/fable-unstick.md +++ /dev/null @@ -1,38 +0,0 @@ -# [M] Hand looping PRs to Fable in half the cases - -## Goal - -A `/quest-iterate` on a PR whose review has run past a threshold of rounds -runs, 50/50 and stable per PR, either on the PR's writer as today or on -Fable 5.2 medium. Its `Implementer:` line records the arm, and `quest usage` -reports the rounds left to merge, findings fixed and refuted, hand-backs, -later culprits, and cost per arm, for the -[routing evidence](/quest/m0/routing-evidence/README.md) readout. - -## Plan - -Follow-up of the 2026-11-13 routing change (user). Theo Browne's -[Opus against Fable video](https://www.youtube.com/watch?v=XtykeVoSmP8) -finds that Opus applies every review finding, so a small PR keeps growing, -while Fable fixes what Opus could not or judges the rest good enough to -merge. Opus low on S quests averaged 4.7 review rounds over 6 reviewed PRs -in `quest usage` since 2026-10-01. - -Agent's choices, for the implementer to revisit: - -- Threshold: past round 4. Most PRs merge by then, so the arms take only - the loops. -- Read the round count from the `## Review` record's tally, as `quest - usage` already does, and pick the arm from a hash of the PR's branch, as - [the writer A/B](/quest/m0/routing-evidence/writer-ab.md) does for - quests. -- The reviewer pick avoids models by exact id, not family, so it avoids only - `claude-fable-5-2` when Fable wrote last; Sol, first in every size's - reviewer list, keeps reviewing such a PR. -- Record the arm on the Implementer line as `(iterate, S, unstick)` or - `(iterate, S, writer)`. - -## Required - -- [usage through token-max](/quest/m0/routing-evidence/usage-token-max.md) - - counts sub-agent writers' cost, without which an arm's cost is undercounted